Lock contention in FileChannel

[Pankaj Gupta <pa...@brightroll.com>]

Hi,

Spent some more time debugging issues with FileChannel. The problem seems
to lock contention reading from FlumeEventQueue:

I see a lot of threads like this:
"SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
[0x00007f821afee000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at
org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
        - waiting to lock <0x0000000518ee4c90> (a
org.apache.flume.channel.file.FlumeEventQueue)
        at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
        at
org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
        at
org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
        at
org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
        at
org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
        at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:662)


I have two file channels and 8 Avro Sinks per file channel. I added more
sinks because they weren't draining fast enough. It seems like they send
the batch then wait for an ack before sending again, thus sends are not
pipelined and having more sinks seemed like a good way of getting some
parallelism.

Here's the full stack trace:
2013-08-13 15:30:32
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02 mixed mode):

"Flume Avro RPC Client Call Invoker 1" daemon prio=10
tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000056d932120> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
at
org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
at $Proxy7.appendBatch(Unknown Source)
at
org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
at
org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Attach Listener" daemon prio=10 tid=0x00007f85fc016000 nid=0x41a1 waiting
on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Flume Avro RPC Client Call Invoker 1" daemon prio=10
tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005194c3340> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Flume Avro RPC Client Call Invoker 1" daemon prio=10
tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005191a16d8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Flume Avro RPC Client Call Invoker 1" daemon prio=10
tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519174e08> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Flume Avro RPC Client Call Invoker 1" daemon prio=10
tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519174e98> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Flume Avro RPC Client Call Invoker 1" daemon prio=10
tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005195f9670> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Flume Avro RPC Client Call Invoker 1" daemon prio=10
tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519174ce8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Flume Avro RPC Client Call Invoker 1" daemon prio=10
tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005191a1918> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Flume Avro RPC Client Call Invoker 1" daemon prio=10
tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005191e7f60> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153 runnable
[0x00007f8210e4d000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000054a291760> (a sun.nio.ch.Util$2)
- locked <0x000000054a2917a8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Flume Avro RPC Client Call Invoker 1" daemon prio=10
tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519174f58> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146 runnable
[0x00007f8210f4e000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
- locked <0x0000000546e5dd28> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118 runnable
[0x00007f821104f000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
- locked <0x0000000544f763c0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111 runnable
[0x00007f8211150000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
- locked <0x000000054473d5a8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110 runnable
[0x00007f8211251000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
- locked <0x00000005437f4f70> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f runnable
[0x00007f8211352000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
- locked <0x00000005422a9018> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e runnable
[0x00007f8211453000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
- locked <0x0000000541063d50> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d runnable
[0x00007f8211554000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
- locked <0x0000000540dc1110> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a runnable
[0x00007f8211655000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
- locked <0x000000053fbbc3b0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109 runnable
[0x00007f8211756000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000053f812878> (a sun.nio.ch.Util$2)
- locked <0x000000053f812860> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd runnable
[0x00007f8211857000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
- locked <0x000000053e6e5570> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Flume Avro RPC Client Call Invoker 1" daemon prio=10
tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519175138> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Flume Avro RPC Client Call Invoker 1" daemon prio=10
tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519174c28> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Flume Avro RPC Client Call Invoker 1" daemon prio=10
tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005191751c8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0 runnable
[0x00007f8211c5b000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
- locked <0x0000000536579db0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Flume Avro RPC Client Call Invoker 1" daemon prio=10
tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005191750a8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1 runnable
[0x00007f8211e5d000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
- locked <0x00000005313943a8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Flume Avro RPC Client Call Invoker 1" daemon prio=10
tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051936f460> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Flume Avro RPC Client Call Invoker 1" daemon prio=10
tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051936dfa8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de runnable
[0x00007f8212160000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
- locked <0x000000052dad4468> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd runnable
[0x00007f8212261000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
- locked <0x000000052c73c5f0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc runnable
[0x00007f8212362000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
- locked <0x000000052b8da998> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db waiting for
monitor entry [0x00007f8212462000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
- waiting to lock <0x0000000518ee4c90> (a
org.apache.flume.channel.file.FlumeEventQueue)
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
at
org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
at
org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
at org.apache.avro.ipc.Responder.respond(Responder.java:149)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Flume Avro RPC Client Call Invoker 1" daemon prio=10
tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051936df18> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Flume Avro RPC Client Call Invoker 1" daemon prio=10
tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000518fe7ae8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8 runnable
[0x00007f8212766000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
- locked <0x00000005263d6ba8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7 runnable
[0x00007f8212867000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
- locked <0x0000000525d1d0e0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6 runnable
[0x00007f8212968000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
- locked <0x0000000525a689d8> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5 runnable
[0x00007f8212a69000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
- locked <0x0000000524e1e7d8> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4 runnable
[0x00007f8212b6a000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
- locked <0x0000000524d437a8> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-36" prio=10 tid=0x00007f85843ad800 nid=0x40d3 runnable
[0x00007f8212c6b000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000524cf4140> (a sun.nio.ch.Util$2)
- locked <0x0000000524cf4128> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000524ab32e8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-35" prio=10 tid=0x00007f85d83a9000 nid=0x40d2 runnable
[0x00007f8212d6c000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005244877c8> (a sun.nio.ch.Util$2)
- locked <0x00000005244877b0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000523f5b660> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-34" prio=10 tid=0x00007f85d838e000 nid=0x40d1 runnable
[0x00007f8212e6d000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000052419e490> (a sun.nio.ch.Util$2)
- locked <0x000000052419e478> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000523f5b6f0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-34" prio=10 tid=0x00007f8588390800 nid=0x40d0 waiting for
monitor entry [0x00007f8212f6d000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
- waiting to lock <0x0000000519111590> (a
org.apache.flume.channel.file.LogFileV3$Writer)
at org.apache.flume.channel.file.Log.put(Log.java:507)
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
at
org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
at
org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
at org.apache.avro.ipc.Responder.respond(Responder.java:149)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-35" prio=10 tid=0x00007f85843ab800 nid=0x40cf runnable
[0x00007f821306f000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000523d40938> (a sun.nio.ch.Util$2)
- locked <0x0000000523d40aa0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000523ecec18> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-37" prio=10 tid=0x00007f85803e3800 nid=0x40ce waiting for
monitor entry [0x00007f821316f000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
- waiting to lock <0x0000000518ee4c90> (a
org.apache.flume.channel.file.FlumeEventQueue)
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
at
org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
at
org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
at org.apache.avro.ipc.Responder.respond(Responder.java:149)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-34" prio=10 tid=0x00007f8584391000 nid=0x40cd runnable
[0x00007f8213271000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000523935540> (a sun.nio.ch.Util$2)
- locked <0x0000000523935528> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000523788ed8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-33" prio=10 tid=0x00007f85d8373800 nid=0x40cc runnable
[0x00007f8213372000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005239ec658> (a sun.nio.ch.Util$2)
- locked <0x00000005239ec640> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000523845410> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-33" prio=10 tid=0x00007f8588375800 nid=0x40cb waiting for
monitor entry [0x00007f8213472000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
- waiting to lock <0x0000000519111590> (a
org.apache.flume.channel.file.LogFileV3$Writer)
at org.apache.flume.channel.file.Log.put(Log.java:507)
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
at
org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
at
org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
at org.apache.avro.ipc.Responder.respond(Responder.java:149)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-33" prio=10 tid=0x00007f8584376000 nid=0x40ca runnable
[0x00007f8213574000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005237e3028> (a sun.nio.ch.Util$2)
- locked <0x00000005237e3010> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000523788e90> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-32" prio=10 tid=0x00007f85d8358800 nid=0x40c9 waiting for
monitor entry [0x00007f8213674000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
- waiting to lock <0x0000000518ee4c90> (a
org.apache.flume.channel.file.FlumeEventQueue)
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
at
org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
at
org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
at org.apache.avro.ipc.Responder.respond(Responder.java:149)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-32" prio=10 tid=0x00007f858835b000 nid=0x40c8 runnable
[0x00007f8213776000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000523343688> (a sun.nio.ch.Util$2)
- locked <0x0000000523343670> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000052305e5c8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-36" prio=10 tid=0x00007f85803c9000 nid=0x40c7 runnable
[0x00007f8213877000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005230fa580> (a sun.nio.ch.Util$2)
- locked <0x00000005230fa568> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000522f5ca50> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-32" prio=10 tid=0x00007f858435b000 nid=0x40c6 runnable
[0x00007f8213978000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005231367a8> (a sun.nio.ch.Util$2)
- locked <0x0000000523136790> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000052305e658> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-31" prio=10 tid=0x00007f85d833e000 nid=0x40c5 runnable
[0x00007f8213a79000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000522db30a0> (a sun.nio.ch.Util$2)
- locked <0x0000000522db3088> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000522bda430> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-31" prio=10 tid=0x00007f8588340000 nid=0x40c4 runnable
[0x00007f8213b7a000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000522c16518> (a sun.nio.ch.Util$2)
- locked <0x0000000522c16500> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000522bd97e8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-30" prio=10 tid=0x00007f85d8323800 nid=0x40c3 runnable
[0x00007f8213c7b000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000522bd1a20> (a sun.nio.ch.Util$2)
- locked <0x0000000522bd1a08> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000522bda3a0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-31" prio=10 tid=0x00007f8584340800 nid=0x40c2 runnable
[0x00007f8213d7c000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005228d1608> (a sun.nio.ch.Util$2)
- locked <0x00000005228d15f0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005228a6aa0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-30" prio=10 tid=0x00007f8588325000 nid=0x40c1 runnable
[0x00007f8213e7d000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000052277b088> (a sun.nio.ch.Util$2)
- locked <0x000000052277b070> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005225ed3b8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-35" prio=10 tid=0x00007f85803ae000 nid=0x40c0 runnable
[0x00007f8213f7e000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000522737380> (a sun.nio.ch.Util$2)
- locked <0x0000000522737368> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005225ed328> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-29" prio=10 tid=0x00007f85d8308800 nid=0x40bf runnable
[0x00007f821407f000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005226034a8> (a sun.nio.ch.Util$2)
- locked <0x0000000522603490> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005225ed298> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-28" prio=10 tid=0x00007f85d82ed800 nid=0x40be runnable
[0x00007f8214180000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000522359408> (a sun.nio.ch.Util$2)
- locked <0x00000005223593f0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005221e7278> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-27" prio=10 tid=0x00007f85d82d2800 nid=0x40bd runnable
[0x00007f8214281000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005222c0478> (a sun.nio.ch.Util$2)
- locked <0x00000005222c0460> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005221e6e40> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-29" prio=10 tid=0x00007f858830a800 nid=0x40bc runnable
[0x00007f8214382000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000521ddf488> (a sun.nio.ch.Util$2)
- locked <0x0000000521ddf470> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000521c88d38> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-30" prio=10 tid=0x00007f8584326000 nid=0x40bb waiting for
monitor entry [0x00007f8214482000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
- waiting to lock <0x0000000519111590> (a
org.apache.flume.channel.file.LogFileV3$Writer)
at org.apache.flume.channel.file.Log.put(Log.java:507)
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
at
org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
at
org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
at org.apache.avro.ipc.Responder.respond(Responder.java:149)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-26" prio=10 tid=0x00007f85d82b8000 nid=0x40ba runnable
[0x00007f8214584000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000521cbcb90> (a sun.nio.ch.Util$2)
- locked <0x0000000521cbcb78> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000521279250> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-28" prio=10 tid=0x00007f85882ef800 nid=0x40b9 runnable
[0x00007f8214685000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000521a1c2d0> (a sun.nio.ch.Util$2)
- locked <0x0000000521a1c2b8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005218f1508> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-34" prio=10 tid=0x00007f8580393800 nid=0x40b8 runnable
[0x00007f8214786000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000052181ce58> (a sun.nio.ch.Util$2)
- locked <0x000000052181ce40> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005212791c0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-29" prio=10 tid=0x00007f858430b800 nid=0x40b7 runnable
[0x00007f8214887000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005219175f0> (a sun.nio.ch.Util$2)
- locked <0x0000000521746090> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000521278e60> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-27" prio=10 tid=0x00007f85882d4800 nid=0x40b6 runnable
[0x00007f8214988000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000052163e9c8> (a sun.nio.ch.Util$2)
- locked <0x000000052163e9b0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000521279130> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-33" prio=10 tid=0x00007f8580378800 nid=0x40b5 runnable
[0x00007f8214a89000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000052152cb38> (a sun.nio.ch.Util$2)
- locked <0x000000052152cb20> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000521278d40> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-28" prio=10 tid=0x00007f85842f0800 nid=0x40b4 runnable
[0x00007f8214b8a000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005213ba3e0> (a sun.nio.ch.Util$2)
- locked <0x00000005213ba3c8> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000521278020> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-27" prio=10 tid=0x00007f85842d5800 nid=0x40b3 runnable
[0x00007f8214c8b000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005213ed660> (a sun.nio.ch.Util$2)
- locked <0x00000005213ed648> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000521278c20> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-26" prio=10 tid=0x00007f85882b9800 nid=0x40b2 runnable
[0x00007f8214d8c000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000052133b7c0> (a sun.nio.ch.Util$2)
- locked <0x000000052133b7a8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005212790a0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-32" prio=10 tid=0x00007f858035d800 nid=0x40af runnable
[0x00007f8214e8d000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000520e53be0> (a sun.nio.ch.Util$2)
- locked <0x0000000520e53bc8> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000520f3bb50> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-25" prio=10 tid=0x00007f85d829d000 nid=0x40ae runnable
[0x00007f8214f8e000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000520e09a30> (a sun.nio.ch.Util$2)
- locked <0x0000000520e09a18> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000520f3b6d0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-31" prio=10 tid=0x00007f8580342800 nid=0x40ad runnable
[0x00007f821508f000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000520ef70e0> (a sun.nio.ch.Util$2)
- locked <0x0000000520ef70c8> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000520f3b208> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-26" prio=10 tid=0x00007f85842ba800 nid=0x40ac runnable
[0x00007f8215190000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000520c9dfc0> (a sun.nio.ch.Util$2)
- locked <0x0000000520c9dfa8> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000520bb3cc0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-25" prio=10 tid=0x00007f85842a0000 nid=0x40ab runnable
[0x00007f8215291000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000520b941a8> (a sun.nio.ch.Util$2)
- locked <0x0000000520b94190> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000520bb30c0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-24" prio=10 tid=0x00007f85d8282800 nid=0x40aa runnable
[0x00007f8215392000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000520952288> (a sun.nio.ch.Util$2)
- locked <0x0000000520952270> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000520356888> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-30" prio=10 tid=0x00007f8580328000 nid=0x40a9 runnable
[0x00007f8215493000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000520a2fa20> (a sun.nio.ch.Util$2)
- locked <0x0000000520a2fa08> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000520983da0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-23" prio=10 tid=0x00007f85d8267800 nid=0x40a8 runnable
[0x00007f8215594000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000052095c470> (a sun.nio.ch.Util$2)
- locked <0x000000052095c458> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000520356840> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-29" prio=10 tid=0x00007f858030d000 nid=0x40a7 runnable
[0x00007f8215695000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000052065b178> (a sun.nio.ch.Util$2)
- locked <0x000000052065b160> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000520357320> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-28" prio=10 tid=0x00007f85802f2800 nid=0x40a6 runnable
[0x00007f8215796000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000052043cad0> (a sun.nio.ch.Util$2)
- locked <0x000000052043cab8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005203567b0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-25" prio=10 tid=0x00007f858829f000 nid=0x40a5 runnable
[0x00007f8215897000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000052041a8a8> (a sun.nio.ch.Util$2)
- locked <0x000000052041a890> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000520356690> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-22" prio=10 tid=0x00007f85d824c800 nid=0x40a4 runnable
[0x00007f8215998000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000052036b380> (a sun.nio.ch.Util$2)
- locked <0x000000052036b368> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000520357290> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-24" prio=10 tid=0x00007f8584285800 nid=0x40a3 runnable
[0x00007f8215a99000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000052007ac40> (a sun.nio.ch.Util$2)
- locked <0x000000052007ac28> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051fe41948> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-27" prio=10 tid=0x00007f85802d8000 nid=0x40a2 runnable
[0x00007f8215b9a000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005201a2ae8> (a sun.nio.ch.Util$2)
- locked <0x00000005201a2ad0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051ffda458> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-21" prio=10 tid=0x00007f85d8232000 nid=0x40a1 waiting for
monitor entry [0x00007f8215c9a000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
- waiting to lock <0x0000000518ee4c90> (a
org.apache.flume.channel.file.FlumeEventQueue)
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
at
org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
at
org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
at org.apache.avro.ipc.Responder.respond(Responder.java:149)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-26" prio=10 tid=0x00007f85802bd800 nid=0x40a0 runnable
[0x00007f8215d9c000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051ff20e78> (a sun.nio.ch.Util$2)
- locked <0x000000051ff20e60> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051fe41900> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-24" prio=10 tid=0x00007f8588284000 nid=0x409f runnable
[0x00007f8215e9d000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051fbc1ca0> (a sun.nio.ch.Util$2)
- locked <0x000000051fbc1c88> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051f9a7978> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-25" prio=10 tid=0x00007f85802a2800 nid=0x409e runnable
[0x00007f8215f9e000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051fa3b5b8> (a sun.nio.ch.Util$2)
- locked <0x000000051fa3b5a0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051f3155b0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-20" prio=10 tid=0x00007f85d8217800 nid=0x409d runnable
[0x00007f821609f000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051f83c838> (a sun.nio.ch.Util$2)
- locked <0x000000051f83c820> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051f3150e8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-23" prio=10 tid=0x00007f8588269800 nid=0x409c runnable
[0x00007f82161a0000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051f78b9b0> (a sun.nio.ch.Util$2)
- locked <0x000000051f78b998> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051f3156d0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-22" prio=10 tid=0x00007f858824e800 nid=0x409b runnable
[0x00007f82162a1000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051f7c0cc8> (a sun.nio.ch.Util$2)
- locked <0x000000051f7c0cb0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051f3151c0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-23" prio=10 tid=0x00007f858426a800 nid=0x409a runnable
[0x00007f82163a2000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051f6065a8> (a sun.nio.ch.Util$2)
- locked <0x000000051f606590> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051f3d9668> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-22" prio=10 tid=0x00007f858424f800 nid=0x4099 runnable
[0x00007f82164a3000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051f56d710> (a sun.nio.ch.Util$2)
- locked <0x000000051f56d6f8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051f314f80> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-21" prio=10 tid=0x00007f8588233800 nid=0x4098 runnable
[0x00007f82165a4000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051f3bcd68> (a sun.nio.ch.Util$2)
- locked <0x000000051f3bcd50> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051f315400> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-19" prio=10 tid=0x00007f85d81fc800 nid=0x4097 runnable
[0x00007f82166a5000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051f53b198> (a sun.nio.ch.Util$2)
- locked <0x000000051f53b180> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051f315760> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-20" prio=10 tid=0x00007f8588218800 nid=0x4096 waiting for
monitor entry [0x00007f82167a5000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
- waiting to lock <0x0000000519111590> (a
org.apache.flume.channel.file.LogFileV3$Writer)
at org.apache.flume.channel.file.Log.put(Log.java:507)
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
at
org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
at
org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
at org.apache.avro.ipc.Responder.respond(Responder.java:149)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-18" prio=10 tid=0x00007f85d81e1800 nid=0x4095 runnable
[0x00007f82168a7000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051ee92ba0> (a sun.nio.ch.Util$2)
- locked <0x000000051ee92b88> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051ed19740> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-19" prio=10 tid=0x00007f85881fd800 nid=0x4094 runnable
[0x00007f82169a8000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051ee5da40> (a sun.nio.ch.Util$2)
- locked <0x000000051ee5da28> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051e77a940> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-17" prio=10 tid=0x00007f85d81c6800 nid=0x4093 runnable
[0x00007f8216aa9000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051ecb1730> (a sun.nio.ch.Util$2)
- locked <0x000000051ecb1718> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051e77a8b0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-21" prio=10 tid=0x00007f8584234800 nid=0x4092 runnable
[0x00007f8216baa000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051e899660> (a sun.nio.ch.Util$2)
- locked <0x000000051e899648> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051e77a310> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-24" prio=10 tid=0x00007f8580287000 nid=0x4091 runnable
[0x00007f8216caa000]
   java.lang.Thread.State: RUNNABLE
at
org.apache.flume.channel.file.FlumeEventQueue.addTail(FlumeEventQueue.java:165)
- locked <0x0000000518ee4c90> (a
org.apache.flume.channel.file.FlumeEventQueue)
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:529)
- locked <0x0000000518ee4c90> (a
org.apache.flume.channel.file.FlumeEventQueue)
at
org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
at org.apache.avro.ipc.Responder.respond(Responder.java:149)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-23" prio=10 tid=0x00007f858026c000 nid=0x4090 runnable
[0x00007f8216dac000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051ea8b6f0> (a sun.nio.ch.Util$2)
- locked <0x000000051ea8b6d8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051e77a820> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-16" prio=10 tid=0x00007f85d81ac000 nid=0x408f runnable
[0x00007f8216ead000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051e7c8808> (a sun.nio.ch.Util$2)
- locked <0x000000051e7c87f0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051e77a280> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-20" prio=10 tid=0x00007f8584219800 nid=0x408e runnable
[0x00007f8216fae000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051e8fffa8> (a sun.nio.ch.Util$2)
- locked <0x000000051e8fff90> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051e77a700> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-22" prio=10 tid=0x00007f8580251000 nid=0x408d runnable
[0x00007f82170af000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051e4dc398> (a sun.nio.ch.Util$2)
- locked <0x000000051e4dc380> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051dcbeef8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-19" prio=10 tid=0x00007f85841fe800 nid=0x408c runnable
[0x00007f82171b0000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051e221520> (a sun.nio.ch.Util$2)
- locked <0x000000051e221508> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051e407860> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-18" prio=10 tid=0x00007f85841e3800 nid=0x408b runnable
[0x00007f82172b1000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051e465298> (a sun.nio.ch.Util$2)
- locked <0x000000051e465280> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051dcbef88> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-18" prio=10 tid=0x00007f85881e3000 nid=0x408a runnable
[0x00007f82173b2000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051dea12e0> (a sun.nio.ch.Util$2)
- locked <0x000000051dea12c8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051cfd4d78> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-15" prio=10 tid=0x00007f85d8191000 nid=0x4089 runnable
[0x00007f82174b3000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051de322b8> (a sun.nio.ch.Util$2)
- locked <0x000000051de322a0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051dcbec28> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-14" prio=10 tid=0x00007f85d8176800 nid=0x4088 runnable
[0x00007f82175b4000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051dea29a8> (a sun.nio.ch.Util$2)
- locked <0x000000051dea2990> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051dcbf018> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-21" prio=10 tid=0x00007f8580236800 nid=0x4087 runnable
[0x00007f82176b5000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051dca95d0> (a sun.nio.ch.Util$2)
- locked <0x000000051dca95b8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051cfd4e08> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-17" prio=10 tid=0x00007f85841c8800 nid=0x4086 runnable
[0x00007f82177b6000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051da8d058> (a sun.nio.ch.Util$2)
- locked <0x000000051da8d040> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051cd693f8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-20" prio=10 tid=0x00007f858021b800 nid=0x4085 runnable
[0x00007f82178b7000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051d95a5b0> (a sun.nio.ch.Util$2)
- locked <0x000000051d95a598> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051cfd4e98> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-16" prio=10 tid=0x00007f85841ad800 nid=0x4083 runnable
[0x00007f82179b8000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051d92a278> (a sun.nio.ch.Util$2)
- locked <0x000000051d92a260> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051cfd4fb8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-13" prio=10 tid=0x00007f85d815b800 nid=0x4082 runnable
[0x00007f8217ab9000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051d734788> (a sun.nio.ch.Util$2)
- locked <0x000000051d734770> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051cd69488> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-19" prio=10 tid=0x00007f8580200800 nid=0x4081 runnable
[0x00007f8217bba000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051d4268e8> (a sun.nio.ch.Util$2)
- locked <0x000000051d4268d0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051d139810> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-15" prio=10 tid=0x00007f8584192800 nid=0x4080 runnable
[0x00007f8217cbb000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051d4cda88> (a sun.nio.ch.Util$2)
- locked <0x000000051d4cda70> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051babcf98> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-17" prio=10 tid=0x00007f85881c8000 nid=0x407f runnable
[0x00007f8217dbc000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051d32ffe8> (a sun.nio.ch.Util$2)
- locked <0x000000051d32ffd0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051cd691b8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-14" prio=10 tid=0x00007f8584177800 nid=0x407e runnable
[0x00007f8217ebd000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051d506c60> (a sun.nio.ch.Util$2)
- locked <0x000000051d506c48> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051cfd5048> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-16" prio=10 tid=0x00007f85881ad000 nid=0x407d runnable
[0x00007f8217fbe000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051d1223e0> (a sun.nio.ch.Util$2)
- locked <0x000000051d1223c8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051c6c7b40> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-15" prio=10 tid=0x00007f8588192800 nid=0x407c runnable
[0x00007f82180bf000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051d146f50> (a sun.nio.ch.Util$2)
- locked <0x000000051d146f38> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051cd69248> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-13" prio=10 tid=0x00007f858415c800 nid=0x407b runnable
[0x00007f82181c0000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051d11d1b8> (a sun.nio.ch.Util$2)
- locked <0x000000051d11d1a0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051c6c7c18> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-12" prio=10 tid=0x00007f85d8141000 nid=0x407a runnable
[0x00007f82182c1000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051cd01a78> (a sun.nio.ch.Util$2)
- locked <0x000000051cd01a60> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051cd695a8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-11" prio=10 tid=0x00007f85d8126800 nid=0x4079 runnable
[0x00007f82183c2000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051cfa6f88> (a sun.nio.ch.Util$2)
- locked <0x000000051cfa6f70> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051ba67370> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-14" prio=10 tid=0x00007f8588177800 nid=0x4078 runnable
[0x00007f82184c3000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051ce5cdd0> (a sun.nio.ch.Util$2)
- locked <0x000000051ce5cdb8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051c6c7bd0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-12" prio=10 tid=0x00007f8584142000 nid=0x4077 runnable
[0x00007f82185c4000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051cc27cb8> (a sun.nio.ch.Util$2)
- locked <0x000000051cc27ca0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051babd0b8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-18" prio=10 tid=0x00007f85801e6000 nid=0x4076 runnable
[0x00007f82186c5000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051cb895c8> (a sun.nio.ch.Util$2)
- locked <0x000000051cc77fd0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051c6c7cf0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-11" prio=10 tid=0x00007f8584127800 nid=0x4075 runnable
[0x00007f82187c6000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051ca540d0> (a sun.nio.ch.Util$2)
- locked <0x000000051caad368> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051c6c7e10> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-13" prio=10 tid=0x00007f858815c800 nid=0x4074 runnable
[0x00007f82188c7000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051c953ab8> (a sun.nio.ch.Util$2)
- locked <0x000000051ca799a8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051babd028> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-17" prio=10 tid=0x00007f85801cb800 nid=0x4073 runnable
[0x00007f82189c8000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051c77c3d8> (a sun.nio.ch.Util$2)
- locked <0x000000051c8137c8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051c3856f0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-10" prio=10 tid=0x00007f858410c800 nid=0x4072 runnable
[0x00007f8218ac9000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051c7dd2b0> (a sun.nio.ch.Util$2)
- locked <0x000000051c613988> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051babcde8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-16" prio=10 tid=0x00007f85801b1000 nid=0x4071 runnable
[0x00007f8218bca000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051c776580> (a sun.nio.ch.Util$2)
- locked <0x000000051c6633c8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051c6c7ea0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-9" prio=10 tid=0x00007f85840f1800 nid=0x4070 runnable
[0x00007f8218ccb000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051c36b550> (a sun.nio.ch.Util$2)
- locked <0x000000051c36b568> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051ba675f8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-8" prio=10 tid=0x00007f85840d6800 nid=0x406f runnable
[0x00007f8218dcc000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051c546b00> (a sun.nio.ch.Util$2)
- locked <0x000000051c40a178> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051ba67760> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-7" prio=10 tid=0x00007f85840bb800 nid=0x406e runnable
[0x00007f8218ecd000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051c2ecf90> (a sun.nio.ch.Util$2)
- locked <0x000000051c29a100> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051ba675b0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-15" prio=10 tid=0x00007f8580196000 nid=0x406d runnable
[0x00007f8218fce000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051c2ee670> (a sun.nio.ch.Util$2)
- locked <0x000000051c2ee688> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051ba67718> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-12" prio=10 tid=0x00007f8588141800 nid=0x406c runnable
[0x00007f82190cf000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051c0bd978> (a sun.nio.ch.Util$2)
- locked <0x000000051c16cc00> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051babcf08> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-6" prio=10 tid=0x00007f85840a0800 nid=0x406b runnable
[0x00007f82191d0000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051c068508> (a sun.nio.ch.Util$2)
- locked <0x000000051bf825f0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051ba676d0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-11" prio=10 tid=0x00007f8588127000 nid=0x406a runnable
[0x00007f82192d1000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051becaa68> (a sun.nio.ch.Util$2)
- locked <0x000000051c03ead0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051babce78> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-10" prio=10 tid=0x00007f858810c800 nid=0x4069 runnable
[0x00007f82193d2000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051be31b98> (a sun.nio.ch.Util$2)
- locked <0x000000051bd56790> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051ba67490> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-14" prio=10 tid=0x00007f858017b000 nid=0x4068 runnable
[0x00007f82194d3000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051bccb8c0> (a sun.nio.ch.Util$2)
- locked <0x000000051bb14420> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051babcd58> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-5" prio=10 tid=0x00007f8584085800 nid=0x4067 runnable
[0x00007f82195d4000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051bd004a8> (a sun.nio.ch.Util$2)
- locked <0x000000051bd49480> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051ba67400> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-4" prio=10 tid=0x00007f858406b000 nid=0x4066 runnable
[0x00007f82196d5000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051ba1e928> (a sun.nio.ch.Util$2)
- locked <0x000000051bccdd00> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051ba66ef0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-9" prio=10 tid=0x00007f85880f1800 nid=0x4065 waiting for
monitor entry [0x00007f82197d5000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
- waiting to lock <0x0000000519111590> (a
org.apache.flume.channel.file.LogFileV3$Writer)
at org.apache.flume.channel.file.Log.put(Log.java:507)
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
at
org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
at
org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
at org.apache.avro.ipc.Responder.respond(Responder.java:149)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-8" prio=10 tid=0x00007f85880d6800 nid=0x4064 runnable
[0x00007f82198d7000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051b5bcca8> (a sun.nio.ch.Util$2)
- locked <0x000000051b7ef8f0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051b5c7cb8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-7" prio=10 tid=0x00007f85880bb800 nid=0x4063 runnable
[0x00007f82199d8000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051b788ce8> (a sun.nio.ch.Util$2)
- locked <0x000000051b788d00> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051b285260> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-13" prio=10 tid=0x00007f8580160800 nid=0x4062 runnable
[0x00007f8219ad9000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051b71a0f8> (a sun.nio.ch.Util$2)
- locked <0x000000051b750e98> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051aed5a70> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-10" prio=10 tid=0x00007f85d810b800 nid=0x4061 runnable
[0x00007f8219bda000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051b517878> (a sun.nio.ch.Util$2)
- locked <0x000000051b5bab30> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051aed5b90> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-9" prio=10 tid=0x00007f85d80f0800 nid=0x4060 runnable
[0x00007f8219cdb000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051b5656e0> (a sun.nio.ch.Util$2)
- locked <0x000000051b5657a0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051b285338> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-8" prio=10 tid=0x00007f85d80d5800 nid=0x405f runnable
[0x00007f8219ddc000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051b3df220> (a sun.nio.ch.Util$2)
- locked <0x000000051b475118> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051aed5c20> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-6" prio=10 tid=0x00007f85880a0800 nid=0x405e runnable
[0x00007f8219edd000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051b11c010> (a sun.nio.ch.Util$2)
- locked <0x000000051b1eab78> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051af91478> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-7" prio=10 tid=0x00007f85d80bb000 nid=0x405d runnable
[0x00007f8219fde000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051b1d36e0> (a sun.nio.ch.Util$2)
- locked <0x000000051b0328c0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051aed5cb0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-12" prio=10 tid=0x00007f8580145800 nid=0x405c runnable
[0x00007f821a0df000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051aeb3070> (a sun.nio.ch.Util$2)
- locked <0x000000051aeb3088> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051a98a820> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-6" prio=10 tid=0x00007f85d80a0000 nid=0x405b runnable
[0x00007f821a1e0000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051aefa920> (a sun.nio.ch.Util$2)
- locked <0x000000051af3ee40> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051ac28810> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-5" prio=10 tid=0x00007f8588086000 nid=0x405a runnable
[0x00007f821a2e1000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051aefe1b0> (a sun.nio.ch.Util$2)
- locked <0x000000051ad2b858> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051aed5e18> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-11" prio=10 tid=0x00007f858012a800 nid=0x4059 runnable
[0x00007f821a3e2000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051ac42588> (a sun.nio.ch.Util$2)
- locked <0x000000051ad05d70> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051a98a7d8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-3" prio=10 tid=0x00007f8584050000 nid=0x4058 runnable
[0x00007f821a4e3000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051ad06100> (a sun.nio.ch.Util$2)
- locked <0x000000051ad47428> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051ac289c0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-10" prio=10 tid=0x00007f858010f800 nid=0x4057 runnable
[0x00007f821a5e4000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051aab5d80> (a sun.nio.ch.Util$2)
- locked <0x000000051ab8ce38> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051a98a748> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-5" prio=10 tid=0x00007f85d8085800 nid=0x4056 runnable
[0x00007f821a6e5000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051a962980> (a sun.nio.ch.Util$2)
- locked <0x000000051a775fb0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051a842098> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-2" prio=10 tid=0x00007f8584035000 nid=0x4055 runnable
[0x00007f821a7e6000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051a817210> (a sun.nio.ch.Util$2)
- locked <0x000000051a7d2b48> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051a841ca8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-4" prio=10 tid=0x00007f85d806b000 nid=0x4054 runnable
[0x00007f821a8e7000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051a903340> (a sun.nio.ch.Util$2)
- locked <0x000000051a903358> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051a42db50> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-9" prio=10 tid=0x00007f85800f4800 nid=0x4053 runnable
[0x00007f821a9e8000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051a4f17d8> (a sun.nio.ch.Util$2)
- locked <0x000000051a37ae08> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051a22e340> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-4" prio=10 tid=0x00007f858806b000 nid=0x4052 runnable
[0x00007f821aae9000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051a4d81e8> (a sun.nio.ch.Util$2)
- locked <0x000000051a616bb0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051a1dfb40> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-3" prio=10 tid=0x00007f85d8050000 nid=0x4051 runnable
[0x00007f821abea000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051a4fb240> (a sun.nio.ch.Util$2)
- locked <0x000000051a3913b8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051a213878> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-2" prio=10 tid=0x00007f85d8035800 nid=0x4050 runnable
[0x00007f821aceb000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051a506248> (a sun.nio.ch.Util$2)
- locked <0x000000051a506308> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051a42dd48> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-8" prio=10 tid=0x00007f85800d9800 nid=0x404f runnable
[0x00007f821adec000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051a2d5170> (a sun.nio.ch.Util$2)
- locked <0x000000051a2d5188> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051a22e2b0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-7" prio=10 tid=0x00007f85800be800 nid=0x404e runnable
[0x00007f821aeed000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051a2d3cb8> (a sun.nio.ch.Util$2)
- locked <0x000000051a2b8310> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051a213758> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
[0x00007f821afee000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
- waiting to lock <0x0000000518ee4c90> (a
org.apache.flume.channel.file.FlumeEventQueue)
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
at
org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
at
org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
at
org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f836c01a000
nid=0x404c runnable [0x00007f821b0ef000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051a045bb0> (a sun.nio.ch.Util$2)
- locked <0x000000051a045bc8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051a000c98> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b376800
nid=0x404b waiting on condition [0x00007f821b1f0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051a045c10> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.NativeThread.current(Native Method)
at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
- locked <0x00000005190ec998> (a java.lang.Object)
at org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
at org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
- locked <0x0000000519111590> (a
org.apache.flume.channel.file.LogFileV3$Writer)
at org.apache.flume.channel.file.Log.take(Log.java:550)
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
at
org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
at
org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
at
org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8354035000
nid=0x4049 runnable [0x00007f821b3f2000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051a045c28> (a sun.nio.ch.Util$2)
- locked <0x000000051a045c40> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051a000ce0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834801a000
nid=0x4048 runnable [0x00007f821b4f3000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051a17bd60> (a sun.nio.ch.Util$2)
- locked <0x000000051a17bd78> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051a180230> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b374800
nid=0x4047 waiting on condition [0x00007f821b5f4000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051a08ba88> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857338a800
nid=0x4046 waiting on condition [0x00007f821b6f5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051a147940> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
tid=0x00007f85f3bff800 nid=0x4045 waiting on condition [0x00007f821b7f6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000056dca65c0> (a
java.util.concurrent.FutureTask$Sync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:227)
at java.util.concurrent.FutureTask.get(FutureTask.java:91)
at
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:323)
at
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:269)
at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:352)
at
org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8344035000
nid=0x4044 runnable [0x00007f821b8f7000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051a184a90> (a sun.nio.ch.Util$2)
- locked <0x000000051a184aa8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051a0bb2e0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bfd800
nid=0x4043 waiting on condition [0x00007f821b9f8000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051a147958> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f833001a000
nid=0x4042 runnable [0x00007f821baf9000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051a188f28> (a sun.nio.ch.Util$2)
- locked <0x000000051a188f40> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519d90758> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8334035000
nid=0x4041 runnable [0x00007f821bbfa000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051a045cd0> (a sun.nio.ch.Util$2)
- locked <0x000000051a045ce8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051a000d28> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b372800
nid=0x4040 waiting on condition [0x00007f821bcfb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519f74590> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573388800
nid=0x403f waiting on condition [0x00007f821bdfc000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519fb0c90> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832c035000
nid=0x403e runnable [0x00007f821befd000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051a045d30> (a sun.nio.ch.Util$2)
- locked <0x000000051a045d48> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051a000d70> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bfb800
nid=0x403d waiting on condition [0x00007f821bffe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519e5d388> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f831801a000
nid=0x403c runnable [0x00007f82341c0000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519ebdb70> (a sun.nio.ch.Util$2)
- locked <0x0000000519ebdb88> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519d3a778> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f831001a000
nid=0x403b runnable [0x00007f82342c1000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519f47f90> (a sun.nio.ch.Util$2)
- locked <0x0000000519f47fa8> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519e11ca8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaf0000
nid=0x403a waiting on condition [0x00007f82343c2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519f73018> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8308035000
nid=0x4039 runnable [0x00007f82344c3000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519f745a8> (a sun.nio.ch.Util$2)
- locked <0x0000000519f745c0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519ca9f78> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573386800
nid=0x4038 waiting on condition [0x00007f82345c4000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519fa5ce0> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bf9800
nid=0x4037 waiting on condition [0x00007f82346c5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519fb0ca8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8300035000
nid=0x4036 runnable [0x00007f82347c6000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519f22d20> (a sun.nio.ch.Util$2)
- locked <0x0000000519f22d38> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519d9f4b8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaee000
nid=0x4035 waiting on condition [0x00007f82348c7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519fb0cc0> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82fc01a000
nid=0x4034 runnable [0x00007f82349c8000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519f74608> (a sun.nio.ch.Util$2)
- locked <0x0000000519f74620> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519ca9fc0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82f001a000
nid=0x4033 runnable [0x00007f8234ac9000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519fb0cd8> (a sun.nio.ch.Util$2)
- locked <0x0000000519fb0cf0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519cadd40> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f44800
nid=0x4032 waiting on condition [0x00007f8234bca000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519ebdbd0> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3377000
nid=0x4031 waiting on condition [0x00007f8234ccb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519f701c0> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82e8035000
nid=0x4030 runnable [0x00007f8234dcc000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519f47ff0> (a sun.nio.ch.Util$2)
- locked <0x0000000519f48008> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519e11cf0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaec000
nid=0x402f waiting on condition [0x00007f8234ecd000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519fa5cf8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
tid=0x00007f857fc38000 nid=0x402e waiting for monitor entry
[0x00007f8234fce000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
- waiting to lock <0x0000000518ee4c90> (a
org.apache.flume.channel.file.FlumeEventQueue)
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
at
org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
at
org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
at
org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82dc01a000
nid=0x402d runnable [0x00007f82350cf000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519ebdbe8> (a sun.nio.ch.Util$2)
- locked <0x0000000519ebdc00> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519d3a7c0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82d001a000
nid=0x402c runnable [0x00007f82351d0000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519fb0d38> (a sun.nio.ch.Util$2)
- locked <0x0000000519fb0d50> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519cadd88> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-144-thread-1" prio=10 tid=0x00007f85d8001000 nid=0x402b runnable
[0x00007f82352d1000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519f481a0> (a sun.nio.ch.Util$2)
- locked <0x0000000519f49040> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519e11d80> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6d3800
nid=0x402a waiting on condition [0x00007f82353d2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519f22d80> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f42800
nid=0x4029 waiting on condition [0x00007f82354d3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519fa5d10> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82cc035000
nid=0x4028 runnable [0x00007f82355d4000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519fb0d98> (a sun.nio.ch.Util$2)
- locked <0x0000000519fb0db0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519caddd0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-3" prio=10 tid=0x00007f8588001000 nid=0x4027 runnable
[0x00007f82356d5000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519ebdc48> (a sun.nio.ch.Util$2)
- locked <0x0000000519ebdc60> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519d3a808> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3375000
nid=0x4026 waiting on condition [0x00007f82357d6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519fa5d28> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82c4035000
nid=0x4025 runnable [0x00007f82358d7000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519f22d98> (a sun.nio.ch.Util$2)
- locked <0x0000000519f22db0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519d9f500> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaea000
nid=0x4024 waiting on condition [0x00007f82359d8000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519fa5d40> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a8035000
nid=0x4023 runnable [0x00007f8235ad9000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519fb0df8> (a sun.nio.ch.Util$2)
- locked <0x0000000519fb0e10> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519cade18> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82b401a000
nid=0x4022 runnable [0x00007f8235bda000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519f73030> (a sun.nio.ch.Util$2)
- locked <0x0000000519f73048> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519e69de8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82ac035000
nid=0x4021 runnable [0x00007f8235cdb000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519f61380> (a sun.nio.ch.Util$2)
- locked <0x0000000519f61398> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005197600f0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6d1800
nid=0x4020 waiting on condition [0x00007f8235ddc000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519f22df8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f40800
nid=0x401f waiting on condition [0x00007f8235edd000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519fa5d58> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3373000
nid=0x401e waiting on condition [0x00007f8235fde000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519ebdca8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a4035000
nid=0x401d runnable [0x00007f82360df000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519fb0e58> (a sun.nio.ch.Util$2)
- locked <0x0000000519fb0e70> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519cade60> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a8000
nid=0x401c waiting on condition [0x00007f82361e0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519f22e10> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-6" prio=10 tid=0x00007f85800a3800 nid=0x401b runnable
[0x00007f82362e1000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519ebde28> (a sun.nio.ch.Util$2)
- locked <0x0000000519de7350> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519d3a898> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828801a000
nid=0x401a runnable [0x00007f82363e2000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519fa5d70> (a sun.nio.ch.Util$2)
- locked <0x0000000519fa5d88> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519d36338> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828c035000
nid=0x4019 runnable [0x00007f82364e3000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519f73090> (a sun.nio.ch.Util$2)
- locked <0x0000000519f730a8> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519e69e30> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8280035000
nid=0x4018 runnable [0x00007f82365e4000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519fb0eb8> (a sun.nio.ch.Util$2)
- locked <0x0000000519e1a7f8> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519cadea8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6cf800
nid=0x4017 waiting on condition [0x00007f82366e5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519f22e28> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f3e800
nid=0x4016 waiting on condition [0x00007f82367e6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519f701d8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3371000
nid=0x4015 waiting on condition [0x00007f82368e7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519fb0f00> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8278035000
nid=0x4014 runnable [0x00007f82369e8000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519e93628> (a sun.nio.ch.Util$2)
- locked <0x0000000519e93640> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519d9f548> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a6000
nid=0x4013 waiting on condition [0x00007f8236ae9000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005199bdaf0> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f827001a000
nid=0x4012 runnable [0x00007f8236bea000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519fb0f18> (a sun.nio.ch.Util$2)
- locked <0x0000000519fb0f30> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519cadef0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6cd800
nid=0x4011 waiting on condition [0x00007f8236ceb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519c47f38> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826c035000
nid=0x4010 runnable [0x00007f8236dec000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519fa5dd0> (a sun.nio.ch.Util$2)
- locked <0x0000000519fa5de8> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519d36380> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826001a000
nid=0x400f runnable [0x00007f8236eed000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519c8c170> (a sun.nio.ch.Util$2)
- locked <0x0000000519c8c188> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519a73d78> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572aeb000
nid=0x400e waiting on condition [0x00007f8236fee000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519c7e570> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2e800
nid=0x400d waiting on condition [0x00007f82370ef000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519c47f50> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8258035000
nid=0x400b runnable [0x00007f82371f0000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519c4b190> (a sun.nio.ch.Util$2)
- locked <0x0000000519c4b1a8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005199eb068> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a4000
nid=0x400a waiting on condition [0x00007f82372f1000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519c47f68> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f825001a000
nid=0x4009 runnable [0x00007f82373f2000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519c33550> (a sun.nio.ch.Util$2)
- locked <0x0000000519c33568> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051996b7b8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e29c000
nid=0x4008 waiting on condition [0x00007f82374f3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519c47f80> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f824c03f800
nid=0x4007 runnable [0x00007f82375f4000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519c7e588> (a sun.nio.ch.Util$2)
- locked <0x0000000519c7e5a0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005199a1910> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8240035000
nid=0x4006 runnable [0x00007f82376f5000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519c47f98> (a sun.nio.ch.Util$2)
- locked <0x0000000519c47fb0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005199536b8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae9000
nid=0x4005 waiting on condition [0x00007f82377f6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519c368c8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2c800
nid=0x4004 waiting on condition [0x00007f82378f7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519c29bf0> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8238035000
nid=0x4003 runnable [0x00007f82379f8000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519bd8198> (a sun.nio.ch.Util$2)
- locked <0x0000000519bd81b0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519b59da0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a2000
nid=0x4002 waiting on condition [0x00007f8237af9000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519c7e5e8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8230035000
nid=0x4001 runnable [0x00007f8237bfa000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519c47ff8> (a sun.nio.ch.Util$2)
- locked <0x0000000519c48010> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519953700> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e29a000
nid=0x4000 waiting on condition [0x00007f8237cfb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519c7e600> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f823001a000
nid=0x3fff runnable [0x00007f8237dfc000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519c7e618> (a sun.nio.ch.Util$2)
- locked <0x0000000519c7e630> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005199a1958> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f823c01a000
nid=0x3ffe runnable [0x00007f8237efd000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519c48058> (a sun.nio.ch.Util$2)
- locked <0x0000000519c48070> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519a6d798> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f823801a000
nid=0x3ffd runnable [0x00007f8237ffe000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519c335b0> (a sun.nio.ch.Util$2)
- locked <0x0000000519c335c8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051996b800> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae7000
nid=0x3ffc waiting on condition [0x00007f84001c0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519c480b8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2a800
nid=0x3ffb waiting on condition [0x00007f84002c1000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519bd81f8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24e800
nid=0x3ffa waiting on condition [0x00007f84003c2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519c29c08> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f824001a000
nid=0x3ff9 runnable [0x00007f84004c3000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519c7e678> (a sun.nio.ch.Util$2)
- locked <0x0000000519c7e690> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005199a19a0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e298000
nid=0x3ff8 waiting on condition [0x00007f84005c4000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519c67408> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-5" prio=10 tid=0x00007f8580088800 nid=0x3ff7 runnable
[0x00007f84006c5000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519c48238> (a sun.nio.ch.Util$2)
- locked <0x0000000519d5e0d0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519a6d828> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f825c01a000
nid=0x3ff6 runnable [0x00007f84007c6000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519c368e0> (a sun.nio.ch.Util$2)
- locked <0x0000000519c368f8> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519953088> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f825801a000
nid=0x3ff5 runnable [0x00007f84008c7000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519c33610> (a sun.nio.ch.Util$2)
- locked <0x0000000519c33628> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051996b848> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826401a000
nid=0x3ff4 runnable [0x00007f84009c8000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519c7e6d8> (a sun.nio.ch.Util$2)
- locked <0x0000000519c7e6f0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005199a19e8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae5000
nid=0x3ff3 waiting on condition [0x00007f8400ac9000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519c730b8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24c800
nid=0x3ff2 waiting on condition [0x00007f8400bca000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519c29c20> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f29000
nid=0x3ff1 waiting on condition [0x00007f8400ccb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519c4b1f0> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826c01a000
nid=0x3ff0 runnable [0x00007f8400dcc000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519bd8210> (a sun.nio.ch.Util$2)
- locked <0x0000000519bd8228> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519b59de8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e296800
nid=0x3fef waiting on condition [0x00007f8400ecd000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519c8c1d0> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f827801a000
nid=0x3fee runnable [0x00007f8400fce000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519c7e738> (a sun.nio.ch.Util$2)
- locked <0x0000000519c7e750> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005199a1a30> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828401a000
nid=0x3fed runnable [0x00007f84010cf000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519c33670> (a sun.nio.ch.Util$2)
- locked <0x0000000519c33688> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051996b890> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828001a000
nid=0x3fec runnable [0x00007f84011d0000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519c29c38> (a sun.nio.ch.Util$2)
- locked <0x0000000519c29c50> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519955c68> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828c01a000
nid=0x3feb runnable [0x00007f84012d1000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519c7e798> (a sun.nio.ch.Util$2)
- locked <0x0000000519c7e7b0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005199533a0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae6000
nid=0x3fea waiting on condition [0x00007f84013d2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519bd8270> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24a800
nid=0x3fe9 waiting on condition [0x00007f84014d3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519c36940> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572273800
nid=0x3fe8 waiting on condition [0x00007f84015d4000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519c8c1e8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de54800
nid=0x3fe7 waiting on condition [0x00007f84016d5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519c67420> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f829001a000
nid=0x3fe6 runnable [0x00007f84017d6000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519c7e7f8> (a sun.nio.ch.Util$2)
- locked <0x0000000519c7e810> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005199533e8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a401a000
nid=0x3fe5 runnable [0x00007f84018d7000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519c336d0> (a sun.nio.ch.Util$2)
- locked <0x0000000519c336e8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051996b8d8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae4000
nid=0x3fe4 waiting on condition [0x00007f84019d8000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519946350> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-4" prio=10 tid=0x00007f858006d800 nid=0x3fe3 runnable
[0x00007f8401ad9000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519bd83d8> (a sun.nio.ch.Util$2)
- locked <0x0000000519f042a0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519b59e78> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82ac01a000
nid=0x3fe2 runnable [0x00007f8401bda000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519c7e858> (a sun.nio.ch.Util$2)
- locked <0x0000000519c7e870> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519953430> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a248800
nid=0x3fe1 waiting on condition [0x00007f8401cdb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519890ec8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a801a000
nid=0x3fe0 runnable [0x00007f8401ddc000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519c8c200> (a sun.nio.ch.Util$2)
- locked <0x0000000519c8c218> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519a73dc0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572271800
nid=0x3fdf waiting on condition [0x00007f8401edd000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005198a3170> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de52800
nid=0x3fde waiting on condition [0x00007f8401fde000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005198b7610> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82b801a000
nid=0x3fdd runnable [0x00007f84020df000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519947f70> (a sun.nio.ch.Util$2)
- locked <0x0000000519947f88> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005197bc938> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82c401a000
nid=0x3fdc runnable [0x00007f84021e0000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519946368> (a sun.nio.ch.Util$2)
- locked <0x0000000519946380> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051983b9b8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82cc01a000
nid=0x3fdb runnable [0x00007f84022e1000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005198a3188> (a sun.nio.ch.Util$2)
- locked <0x00000005198a31a0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005197300e0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae2000
nid=0x3fda waiting on condition [0x00007f84023e2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005199463c8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857226f800
nid=0x3fd9 waiting on condition [0x00007f84024e3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519792380> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82c801a000
nid=0x3fd8 runnable [0x00007f84025e4000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519947fd0> (a sun.nio.ch.Util$2)
- locked <0x0000000519947fe8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005197bc980> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579597000
nid=0x3fd7 waiting on condition [0x00007f84026e5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519488cc0> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de50800
nid=0x3fd6 waiting on condition [0x00007f84027e6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005198cc640> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82d801a000
nid=0x3fd5 runnable [0x00007f84028e7000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005199463e0> (a sun.nio.ch.Util$2)
- locked <0x00000005199463f8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051983ba00> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82e401a000
nid=0x3fd4 runnable [0x00007f84029e8000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005198f7fc8> (a sun.nio.ch.Util$2)
- locked <0x00000005198f7fe0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005194d0160> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82ec01a000
nid=0x3fd3 runnable [0x00007f8402ae9000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005198a31e8> (a sun.nio.ch.Util$2)
- locked <0x00000005198a3200> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519730128> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae0800
nid=0x3fd2 waiting on condition [0x00007f8402bea000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519948030> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857226d800
nid=0x3fd1 waiting on condition [0x00007f8402ceb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519946440> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82e801a000
nid=0x3fd0 runnable [0x00007f8402dec000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519890ee0> (a sun.nio.ch.Util$2)
- locked <0x0000000519890ef8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005196f56f8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579595000
nid=0x3fcf waiting on condition [0x00007f8402eed000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005198b7628> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de4e800
nid=0x3fce waiting on condition [0x00007f8402fee000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005198a3248> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830401a000
nid=0x3fcd runnable [0x00007f84030ef000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519946458> (a sun.nio.ch.Util$2)
- locked <0x0000000519946470> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051983ba48> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830001a000
nid=0x3fcc runnable [0x00007f84031f0000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519937b88> (a sun.nio.ch.Util$2)
- locked <0x0000000519937ba0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519887cb8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830c01a000
nid=0x3fcb runnable [0x00007f84032f1000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519948048> (a sun.nio.ch.Util$2)
- locked <0x0000000519948060> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005197bc9c8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830801a000
nid=0x3fca runnable [0x00007f84033f2000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005199464b8> (a sun.nio.ch.Util$2)
- locked <0x00000005199464d0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051983ba90> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e2b000
nid=0x3fc9 waiting on condition [0x00007f84034f3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519943980> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1595000
nid=0x3fc8 waiting on condition [0x00007f84035f4000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005198a3260> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857bff5800
nid=0x3fc7 waiting on condition [0x00007f84036f5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519901138> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da0c000
nid=0x3fc6 waiting on condition [0x00007f84037f6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005199480a8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832001a000
nid=0x3fc5 runnable [0x00007f84038f7000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519946518> (a sun.nio.ch.Util$2)
- locked <0x0000000519946530> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051983bad8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832c01a000
nid=0x3fc4 runnable [0x00007f84039f8000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005198a3278> (a sun.nio.ch.Util$2)
- locked <0x00000005198a3290> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519730170> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832801a000
nid=0x3fc3 runnable [0x00007f8403af9000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005198f8028> (a sun.nio.ch.Util$2)
- locked <0x00000005198f8040> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005194d01a8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f833401a000
nid=0x3fc2 runnable [0x00007f8403bfa000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005198b7640> (a sun.nio.ch.Util$2)
- locked <0x00000005198b7658> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005196f3480> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e29000
nid=0x3fc1 waiting on condition [0x00007f8403cfb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519946578> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1593000
nid=0x3fc0 waiting on condition [0x00007f8403dfc000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005199480c0> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579592000
nid=0x3fbf waiting on condition [0x00007f8403efd000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519943998> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da0a800
nid=0x3fbe waiting on condition [0x00007f84401c0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519946590> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834401a000
nid=0x3fbd runnable [0x00007f84402c1000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519903410> (a sun.nio.ch.Util$2)
- locked <0x0000000519903428> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005191a1498> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834001a000
nid=0x3fbc runnable [0x00007f84403c2000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519943ef0> (a sun.nio.ch.Util$2)
- locked <0x0000000519943f08> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005196f5740> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834c01a000
nid=0x3fbb runnable [0x00007f84404c3000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005199497b0> (a sun.nio.ch.Util$2)
- locked <0x00000005199497c8> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519685f60> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573ff1800
nid=0x3fba waiting on condition [0x00007f84405c4000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005198f8088> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1591800
nid=0x3fb9 waiting on condition [0x00007f84406c5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005198a32d8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579591000
nid=0x3fb8 waiting on condition [0x00007f84407c6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005199480d8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f835401a000
nid=0x3fb7 runnable [0x00007f84408c7000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005199465a8> (a sun.nio.ch.Util$2)
- locked <0x00000005199465c0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051983bb20> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da08800
nid=0x3fb6 waiting on condition [0x00007f84409c8000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005199439b0> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"pool-142-thread-1" prio=10 tid=0x00007f8584001000 nid=0x3fb5 runnable
[0x00007f8440ac9000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005198b77f0> (a sun.nio.ch.Util$2)
- locked <0x0000000519a00960> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519952e48> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f837401a000
nid=0x3fb4 runnable [0x00007f8440bca000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519937be8> (a sun.nio.ch.Util$2)
- locked <0x0000000519937c00> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519887d00> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f836801a000
nid=0x3fb3 runnable [0x00007f8440ccb000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519943f50> (a sun.nio.ch.Util$2)
- locked <0x0000000519943f68> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005196f5788> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f837001a000
nid=0x3fb2 runnable [0x00007f8440dcc000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519901150> (a sun.nio.ch.Util$2)
- locked <0x0000000519901168> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519106100> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f158f800
nid=0x3fb1 waiting on condition [0x00007f8440ecd000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519946608> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e26800
nid=0x3fb0 waiting on condition [0x00007f8440fce000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005198a32f0> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913e000
nid=0x3faf waiting on condition [0x00007f84410cf000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005199439c8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f837801a000
nid=0x3fae runnable [0x00007f84411d0000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005199480f0> (a sun.nio.ch.Util$2)
- locked <0x0000000519948108> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005197bca10> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da07000
nid=0x3fad waiting on condition [0x00007f84412d1000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519943fb0> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f838801a000
nid=0x3fac runnable [0x00007f84413d2000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005198f80a0> (a sun.nio.ch.Util$2)
- locked <0x00000005198f80b8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005194d01f0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f839401a000
nid=0x3fab runnable [0x00007f84414d3000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519946620> (a sun.nio.ch.Util$2)
- locked <0x0000000519946638> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051983bb68> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f839001a000
nid=0x3faa runnable [0x00007f84415d4000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519937c48> (a sun.nio.ch.Util$2)
- locked <0x0000000519937c60> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519887d48> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d0d000
nid=0x3fa9 waiting on condition [0x00007f84416d5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519654310> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913c000
nid=0x3fa8 waiting on condition [0x00007f84417d6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005191b54d8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571164000
nid=0x3fa7 waiting on condition [0x00007f84418d7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519637a20> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83a801a000
nid=0x3fa6 runnable [0x00007f84419d8000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005195f76c8> (a sun.nio.ch.Util$2)
- locked <0x00000005196430c0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005194ecaf0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83b401a000
nid=0x3fa5 runnable [0x00007f8441ad9000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051965f960> (a sun.nio.ch.Util$2)
- locked <0x000000051965f978> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519104648> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83b001a000
nid=0x3fa4 runnable [0x00007f8441bda000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519655e28> (a sun.nio.ch.Util$2)
- locked <0x0000000519655e40> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519104ba0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-3" prio=10 tid=0x00007f8580052800 nid=0x3fa3 runnable
[0x00007f8441cdb000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519637b10> (a sun.nio.ch.Util$2)
- locked <0x0000000519694210> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051916e518> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83bc01a000
nid=0x3fa2 runnable [0x00007f8441ddc000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519597678> (a sun.nio.ch.Util$2)
- locked <0x0000000519597690> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519106388> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c4800
nid=0x3fa1 waiting on condition [0x00007f8441edd000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519654328> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d0b000
nid=0x3fa0 waiting on condition [0x00007f8441fde000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519676ad0> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571162000
nid=0x3f9f waiting on condition [0x00007f84420df000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519655618> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913a000
nid=0x3f9e waiting on condition [0x00007f84421e0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005194ae248> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-2" prio=10 tid=0x00007f8580001000 nid=0x3f9d runnable
[0x00007f84422e1000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519654340> (a sun.nio.ch.Util$2)
- locked <0x0000000519654358> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519105380> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83cc01a000
nid=0x3f9c runnable [0x00007f84423e2000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519655e88> (a sun.nio.ch.Util$2)
- locked <0x0000000519655ea0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519104be8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83d001a000
nid=0x3f9b runnable [0x00007f84424e3000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005194e1350> (a sun.nio.ch.Util$2)
- locked <0x00000005194e1368> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005191041c8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c3000
nid=0x3f9a waiting on condition [0x00007f84425e4000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051967caf8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83d801a000
nid=0x3f99 runnable [0x00007f84426e5000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519655ee8> (a sun.nio.ch.Util$2)
- locked <0x0000000519655f00> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519104c30> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83dc01a000
nid=0x3f98 runnable [0x00007f84427e6000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005196543a0> (a sun.nio.ch.Util$2)
- locked <0x00000005196543b8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005191053c8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d09800
nid=0x3f97 waiting on condition [0x00007f84428e7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519655630> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579138000
nid=0x3f96 waiting on condition [0x00007f84429e8000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005196592c8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571160000
nid=0x3f95 waiting on condition [0x00007f8442ae9000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519655f48> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f840801a000
nid=0x3f94 runnable [0x00007f8442bea000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519655648> (a sun.nio.ch.Util$2)
- locked <0x0000000519655660> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051914b290> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83e801a000
nid=0x3f93 runnable [0x00007f8442ceb000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519643108> (a sun.nio.ch.Util$2)
- locked <0x0000000519643120> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005194ecb38> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d07800
nid=0x3f92 waiting on condition [0x00007f8442dec000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519658428> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83f801a000
nid=0x3f91 runnable [0x00007f8442eed000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005194c9908> (a sun.nio.ch.Util$2)
- locked <0x00000005194c9920> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519511968> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83fc01a000
nid=0x3f90 runnable [0x00007f8442fee000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519643168> (a sun.nio.ch.Util$2)
- locked <0x0000000519643180> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005194ecb80> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578465000
nid=0x3f8f waiting on condition [0x00007f8403ffe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519654400> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857115e800
nid=0x3f8e waiting on condition [0x00007f84430ef000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005196556a8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c1000
nid=0x3f8d waiting on condition [0x00007f84431f0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519654418> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f840c01a000
nid=0x3f8c runnable [0x00007f84432f1000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005196431c8> (a sun.nio.ch.Util$2)
- locked <0x00000005196431e0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005194ecbc8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f841801a000
nid=0x3f8b runnable [0x00007f84433f2000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519655f60> (a sun.nio.ch.Util$2)
- locked <0x0000000519655f78> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519104c78> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b4000
nid=0x3f8a waiting on condition [0x00007f84434f3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005196721e8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5bf000
nid=0x3f89 waiting on condition [0x00007f84435f4000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519654430> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f841c01a000
nid=0x3f88 runnable [0x00007f84436f5000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005196556c0> (a sun.nio.ch.Util$2)
- locked <0x00000005196556d8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051914b2d8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f842801a000
nid=0x3f87 runnable [0x00007f84437f6000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519643228> (a sun.nio.ch.Util$2)
- locked <0x0000000519643240> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005194ecc10> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b2000
nid=0x3f86 waiting on condition [0x00007f84438f7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519654448> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5bd800
nid=0x3f85 waiting on condition [0x00007f84439f8000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051967cb10> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f843001a000
nid=0x3f84 runnable [0x00007f8443af9000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519655fc0> (a sun.nio.ch.Util$2)
- locked <0x0000000519655fd8> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519104cc0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b0800
nid=0x3f83 waiting on condition [0x00007f8443bfa000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005196592e0> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f843801a000
nid=0x3f82 runnable [0x00007f8443cfb000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519643288> (a sun.nio.ch.Util$2)
- locked <0x00000005196432a0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005194ecc58> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d17b000
nid=0x3f81 waiting on condition [0x00007f8443dfc000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051921afd8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f843c01a000
nid=0x3f80 runnable [0x00007f8443efd000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519654460> (a sun.nio.ch.Util$2)
- locked <0x0000000519654478> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519105410> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08ae800
nid=0x3f7f waiting on condition [0x00007f8443ffe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519676ae8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f844401a000
nid=0x3f7e runnable [0x00007f85441a6000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051967cb28> (a sun.nio.ch.Util$2)
- locked <0x000000051967cb40> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005194745e0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d179000
nid=0x3f7d waiting on condition [0x00007f85442a7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005196432e8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f845801a000
nid=0x3f7c runnable [0x00007f85443a8000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519658440> (a sun.nio.ch.Util$2)
- locked <0x00000005196583f8> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519349128> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f845401a000
nid=0x3f7b runnable [0x00007f85444a9000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519676aa0> (a sun.nio.ch.Util$2)
- locked <0x0000000519676ab8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051921b020> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08ad800
nid=0x3f7a waiting on condition [0x00007f85446c5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519659298> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578463000
nid=0x3f79 waiting on condition [0x00007f85447c6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005194a3568> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f845c01a000
nid=0x3f78 runnable [0x00007f85445aa000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005196544c0> (a sun.nio.ch.Util$2)
- locked <0x00000005196544d8> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519105458> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d177000
nid=0x3f77 waiting on condition [0x00007f85448c7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519656020> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f846401a000
nid=0x3f76 runnable [0x00007f85449c8000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519655720> (a sun.nio.ch.Util$2)
- locked <0x0000000519655738> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519105f98> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857115c800
nid=0x3f75 waiting on condition [0x00007f8544ac9000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519402f58> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f847801a000
nid=0x3f74 runnable [0x00007f8544bca000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519654520> (a sun.nio.ch.Util$2)
- locked <0x0000000519654538> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005191054a0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f847401a000
nid=0x3f73 runnable [0x00007f8544ccb000]
   java.lang.Thread.State: RUNNABLE
at java.util.HashMap.newKeyIterator(HashMap.java:840)
at java.util.HashMap$KeySet.iterator(HashMap.java:874)
at java.util.HashSet.iterator(HashSet.java:153)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
- locked <0x0000000519656248> (a java.util.HashSet)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519656038> (a sun.nio.ch.Util$2)
- locked <0x0000000519656050> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519104d08> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0015800
nid=0x3f72 waiting on condition [0x00007f8544dcc000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051941d010> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578461000
nid=0x3f71 waiting on condition [0x00007f8544ecd000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519477938> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f847c01a000
nid=0x3f70 runnable [0x00007f8544fce000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051942aa08> (a sun.nio.ch.Util$2)
- locked <0x000000051942aa20> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519349050> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d175800
nid=0x3f6f waiting on condition [0x00007f85450cf000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519470c60> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f848401a000
nid=0x3f6e runnable [0x00007f85451d0000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519445fa0> (a sun.nio.ch.Util$2)
- locked <0x0000000519445fb8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005194679d0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570456800
nid=0x3f6d waiting on condition [0x00007f85452d1000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519419230> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f849801a000
nid=0x3f6c runnable [0x00007f85453d2000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519470c78> (a sun.nio.ch.Util$2)
- locked <0x0000000519470c90> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005194743a0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f849401a000
nid=0x3f6b runnable [0x00007f85454d3000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051945beb8> (a sun.nio.ch.Util$2)
- locked <0x000000051945bed0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519196000> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0013800
nid=0x3f6a waiting on condition [0x00007f85455d4000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519470cd8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857845f000
nid=0x3f69 waiting on condition [0x00007f85456d5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051941d1c0> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f849c01a000
nid=0x3f68 runnable [0x00007f85457d6000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005194583b8> (a sun.nio.ch.Util$2)
- locked <0x00000005194583d0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005194743e8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c488000
nid=0x3f67 waiting on condition [0x00007f85458d7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519424e80> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84a401a000
nid=0x3f66 runnable [0x00007f85459d8000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051941d1d8> (a sun.nio.ch.Util$2)
- locked <0x000000051941d1f0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519467a18> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570455000
nid=0x3f65 waiting on condition [0x00007f8545ad9000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051945bf18> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84b801a000
nid=0x3f64 runnable [0x00007f8545bda000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519419248> (a sun.nio.ch.Util$2)
- locked <0x0000000519419200> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519431ee0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84c001a000
nid=0x3f63 runnable [0x00007f8545cdb000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051941d238> (a sun.nio.ch.Util$2)
- locked <0x000000051941d250> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519467a60> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0012000
nid=0x3f62 waiting on condition [0x00007f8545ddc000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519458418> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84bc01a000
nid=0x3f61 runnable [0x00007f8545edd000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051945bf30> (a sun.nio.ch.Util$2)
- locked <0x000000051945bf48> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519196048> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857845e000
nid=0x3f60 waiting on condition [0x00007f8545fde000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519479f10> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c486000
nid=0x3f5f waiting on condition [0x00007f85460df000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051941d298> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84c401a000
nid=0x3f5e runnable [0x00007f85461e0000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051942a9a8> (a sun.nio.ch.Util$2)
- locked <0x000000051942a9c0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519349098> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570453800
nid=0x3f5d waiting on condition [0x00007f85462e1000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051945bf90> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84cc01a000
nid=0x3f5c runnable [0x00007f85463e2000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519458430> (a sun.nio.ch.Util$2)
- locked <0x0000000519458448> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519474430> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c484800
nid=0x3f5b waiting on condition [0x00007f85464e3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051941d2b0> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84d401a000
nid=0x3f5a runnable [0x00007f85465e4000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005194191e8> (a sun.nio.ch.Util$2)
- locked <0x000000051936a4e0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519431f28> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570450800
nid=0x3f59 waiting on condition [0x00007f85466e5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051943ef60> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84dc01a000
nid=0x3f58 runnable [0x00007f85467e6000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051942a948> (a sun.nio.ch.Util$2)
- locked <0x000000051942a960> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005193490e0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c482800
nid=0x3f57 waiting on condition [0x00007f85468e7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051945bfa8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84e401a000
nid=0x3f56 runnable [0x00007f85469e8000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051941d2c8> (a sun.nio.ch.Util$2)
- locked <0x000000051941d2e0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519467aa8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857044f000
nid=0x3f55 waiting on condition [0x00007f8546ae9000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051945bfc0> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84f801a000
nid=0x3f54 runnable [0x00007f8546bea000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051936a4c8> (a sun.nio.ch.Util$2)
- locked <0x000000051936a480> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519431f70> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84f401a000
nid=0x3f53 runnable [0x00007f8546ceb000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519458490> (a sun.nio.ch.Util$2)
- locked <0x00000005194584a8> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519474478> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578008000
nid=0x3f52 waiting on condition [0x00007f8546dec000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051942a8e8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c00d000
nid=0x3f51 waiting on condition [0x00007f8546eed000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519447098> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84fc01a000
nid=0x3f50 runnable [0x00007f8546fee000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051941d328> (a sun.nio.ch.Util$2)
- locked <0x000000051941d340> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519467af0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857044d000
nid=0x3f4f waiting on condition [0x00007f85470ef000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051945bfd8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f851001a000
nid=0x3f4e runnable [0x00007f85471f0000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519424e98> (a sun.nio.ch.Util$2)
- locked <0x0000000519424e50> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519380d20> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f850c01a000
nid=0x3f4d runnable [0x00007f85472f1000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005194584f0> (a sun.nio.ch.Util$2)
- locked <0x0000000519458508> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005194744c0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578006800
nid=0x3f4c waiting on condition [0x00007f85473f2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051945bff0> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c01d800
nid=0x3f4b waiting on condition [0x00007f85474f3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051936a468> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f851401a000
nid=0x3f4a runnable [0x00007f85475f4000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051941d388> (a sun.nio.ch.Util$2)
- locked <0x000000051941d3a0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519467b38> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857045b800
nid=0x3f49 waiting on condition [0x00007f85476f5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051942a900> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f853801a000
nid=0x3f48 runnable [0x00007f85477f6000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519479f28> (a sun.nio.ch.Util$2)
- locked <0x0000000519479ee0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519203068> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f853001f800
nid=0x3f47 runnable [0x00007f85478f7000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519477950> (a sun.nio.ch.Util$2)
- locked <0x0000000519477968> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051922faf0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f852c01f000
nid=0x3f46 runnable [0x00007f85479f8000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005194353f8> (a sun.nio.ch.Util$2)
- locked <0x0000000519435410> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000051921b0f8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f853401a000
nid=0x3f45 runnable [0x00007f8547af9000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051945c008> (a sun.nio.ch.Util$2)
- locked <0x000000051945c020> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000519196090> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578005800
nid=0x3f44 waiting on condition [0x00007f8547bfa000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005190de788> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f001a800
nid=0x3f43 waiting on condition [0x00007f8547cfb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000518fae040> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c022000
nid=0x3f42 waiting on condition [0x00007f8547dfc000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000518cafb20> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570460000
nid=0x3f41 waiting on condition [0x00007f8547efd000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005190db9c8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-2" prio=10 tid=0x00007f8588036000 nid=0x3f40 runnable
[0x00007f8547ffe000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005190ee0e8> (a sun.nio.ch.Util$2)
- locked <0x00000005191f0be8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005190e1d18> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-140-thread-1" prio=10 tid=0x00007f858801b800 nid=0x3f3f runnable
[0x00007f858c18a000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000518f16658> (a sun.nio.ch.Util$2)
- locked <0x000000051941f820> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005190edc80> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-139-thread-1" prio=10 tid=0x00007f8580002800 nid=0x3f3e runnable
[0x00007f858c28b000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x000000051913eb50> (a sun.nio.ch.Util$2)
- locked <0x000000051913eca0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000518c9c4e8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-148-thread-1" prio=10 tid=0x00007f856844c000 nid=0x3f3d waiting on
condition [0x00007f858c38c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000005190cfe98> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"pool-146-thread-1" prio=10 tid=0x00007f85dc006800 nid=0x3f3c waiting on
condition [0x00007f858c48d000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519018558> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"pool-147-thread-1" prio=10 tid=0x00007f85e82b1000 nid=0x3f3b waiting on
condition [0x00007f858c58e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051903b488> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"pool-145-thread-1" prio=10 tid=0x00007f8574027000 nid=0x3f3a waiting on
condition [0x00007f858c68f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051903b998> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"pool-141-thread-1" prio=10 tid=0x00007f85e8159000 nid=0x3f39 runnable
[0x00007f861cd8e000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005190db9e0> (a sun.nio.ch.Util$2)
- locked <0x00000005190db998> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000518c9c020> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-138-thread-1" prio=10 tid=0x00007f856844a800 nid=0x3f38 runnable
[0x00007f861cf90000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000005190e1b68> (a sun.nio.ch.Util$2)
- locked <0x00000005190e1b80> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000518c9c608> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-143-thread-1" prio=10 tid=0x00007f85dc003000 nid=0x3f37 runnable
[0x00007f861d091000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519018fa8> (a sun.nio.ch.Util$2)
- locked <0x0000000519018fc0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005190e2228> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"pool-137-thread-1" prio=10 tid=0x00007f8574007000 nid=0x3f36 runnable
[0x00007f861ce8f000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x0000000519003dc8> (a sun.nio.ch.Util$2)
- locked <0x0000000519003de0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005190ea1a8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"1407844613@qtp-1263282395-1 - Acceptor0 SocketConnector@0.0.0.0:54321"
prio=10 tid=0x00007f85e404a800 nid=0x3f35 runnable [0x00007f858c790000]
   java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
- locked <0x00000005190e2720> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at org.mortbay.jetty.bio.SocketConnector.accept(SocketConnector.java:99)
at
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

"208924888@qtp-1263282395-0" prio=10 tid=0x00007f85e4021800 nid=0x3f34 in
Object.wait() [0x00007f861c15b000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000005190d0528> (a
org.mortbay.thread.QueuedThreadPool$PoolThread)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
- locked <0x00000005190d0528> (a
org.mortbay.thread.QueuedThreadPool$PoolThread)

"lifecycleSupervisor-1-9" prio=10 tid=0x00007f85e401a800 nid=0x3f33 waiting
on condition [0x00007f861c687000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000518c85670> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"lifecycleSupervisor-1-8" prio=10 tid=0x00007f85e4018800 nid=0x3f32 waiting
on condition [0x00007f861c788000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000518c85670> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"lifecycleSupervisor-1-7" prio=10 tid=0x00007f85e4016800 nid=0x3f31 waiting
on condition [0x00007f861c889000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000518c85670> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"lifecycleSupervisor-1-6" prio=10 tid=0x00007f85e4014800 nid=0x3f30 waiting
on condition [0x00007f861c98a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000518c85670> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"lifecycleSupervisor-1-5" prio=10 tid=0x00007f85e4013000 nid=0x3f2f waiting
on condition [0x00007f861ca8b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000518c85670> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"lifecycleSupervisor-1-4" prio=10 tid=0x00007f85e4028000 nid=0x3f2e waiting
on condition [0x00007f861cb8c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000518c85670> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"lifecycleSupervisor-1-3" prio=10 tid=0x00007f85e4026800 nid=0x3f2d waiting
on condition [0x00007f861cc8d000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000518c85670> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Log-BackgroundWorker-ch1" prio=10 tid=0x00007f85dc001000 nid=0x3f27
waiting on condition [0x00007f861d299000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000519017718> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Log-BackgroundWorker-ch2" prio=10 tid=0x00007f85e80ca000 nid=0x3f26
waiting on condition [0x00007f861d39a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x000000051903b4e8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"lifecycleSupervisor-1-2" prio=10 tid=0x00007f85e4025000 nid=0x3f25 waiting
on condition [0x00007f861d6ba000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000518c85670> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"lifecycleSupervisor-1-1" prio=10 tid=0x00007f85e400e800 nid=0x3f24 waiting
on condition [0x00007f861d7bb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000518c85670> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"conf-file-poller-0" prio=10 tid=0x00007f85f0001000 nid=0x3f23 waiting on
condition [0x00007f861d8bc000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000518ec72f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"DestroyJavaVM" prio=10 tid=0x00007f86a800c800 nid=0x3efa waiting on
condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"lifecycleSupervisor-1-0" prio=10 tid=0x00007f86a89f5800 nid=0x3f22 waiting
on condition [0x00007f861d9bd000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0000000518c85670> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

"Low Memory Detector" daemon prio=10 tid=0x00007f86a831d800 nid=0x3f20
runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=10 tid=0x00007f86a831b000 nid=0x3f1f
waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=10 tid=0x00007f86a8319000 nid=0x3f1e
waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00007f86a8316800 nid=0x3f1d
runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (Concurrent GC)" daemon prio=10
tid=0x00007f86a8314800 nid=0x3f1c waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00007f86a82f9800 nid=0x3f1b in
Object.wait() [0x00007f861e3eb000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000518c843c0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x0000000518c843c0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x00007f86a82f7800 nid=0x3f1a in
Object.wait() [0x00007f861e4ec000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000518c80860> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x0000000518c80860> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=10 tid=0x00007f86a82f1000 nid=0x3f19 runnable

"Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f86a801b800
nid=0x3efb runnable

"Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f86a801d000
nid=0x3efc runnable

"Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f86a801f000
nid=0x3efd runnable

"Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f86a8021000
nid=0x3efe runnable

"Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x00007f86a8022800
nid=0x3eff runnable

"Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x00007f86a8024800
nid=0x3f00 runnable

"Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x00007f86a8026000
nid=0x3f01 runnable

"Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x00007f86a8028000
nid=0x3f02 runnable

"Gang worker#8 (Parallel GC Threads)" prio=10 tid=0x00007f86a802a000
nid=0x3f03 runnable

"Gang worker#9 (Parallel GC Threads)" prio=10 tid=0x00007f86a802b800
nid=0x3f04 runnable

"Gang worker#10 (Parallel GC Threads)" prio=10 tid=0x00007f86a802d800
nid=0x3f05 runnable

"Gang worker#11 (Parallel GC Threads)" prio=10 tid=0x00007f86a802f800
nid=0x3f06 runnable

"Gang worker#12 (Parallel GC Threads)" prio=10 tid=0x00007f86a8031000
nid=0x3f07 runnable

"Gang worker#13 (Parallel GC Threads)" prio=10 tid=0x00007f86a8033000
nid=0x3f08 runnable

"Gang worker#14 (Parallel GC Threads)" prio=10 tid=0x00007f86a8035000
nid=0x3f09 runnable

"Gang worker#15 (Parallel GC Threads)" prio=10 tid=0x00007f86a8036800
nid=0x3f0a runnable

"Gang worker#16 (Parallel GC Threads)" prio=10 tid=0x00007f86a8038800
nid=0x3f0b runnable

"Gang worker#17 (Parallel GC Threads)" prio=10 tid=0x00007f86a803a000
nid=0x3f0c runnable

"Gang worker#18 (Parallel GC Threads)" prio=10 tid=0x00007f86a803c000
nid=0x3f0d runnable

"Gang worker#19 (Parallel GC Threads)" prio=10 tid=0x00007f86a803e000
nid=0x3f0e runnable

"Gang worker#20 (Parallel GC Threads)" prio=10 tid=0x00007f86a803f800
nid=0x3f0f runnable

"Gang worker#21 (Parallel GC Threads)" prio=10 tid=0x00007f86a8041800
nid=0x3f10 runnable

"Gang worker#22 (Parallel GC Threads)" prio=10 tid=0x00007f86a8043800
nid=0x3f11 runnable

"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f86a82aa000 nid=0x3f18
runnable
"Gang worker#0 (Parallel CMS Threads)" prio=10 tid=0x00007f86a829e800
nid=0x3f12 runnable

"Gang worker#1 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a0000
nid=0x3f13 runnable

"Gang worker#2 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a2000
nid=0x3f14 runnable

"Gang worker#3 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a3800
nid=0x3f15 runnable

"Gang worker#4 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a5800
nid=0x3f16 runnable

"Gang worker#5 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a7800
nid=0x3f17 runnable

"VM Periodic Task Thread" prio=10 tid=0x00007f86a8329000 nid=0x3f21 waiting
on condition

JNI global references: 1749



-- 


*P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com

Pankaj Gupta | Software Engineer

*BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com


United States | Canada | United Kingdom | Germany


We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
!

Re: Lock contention in FileChannel

[Pankaj Gupta <pa...@brightroll.com>]

Replacing FileChannel with Memory Channel I see way better performance,
which confirms that there are no other bottlenecks in the rest of the
subsystem i.e. the AvroSinks can accept and transfer data to the next box
if the channel allows them to read faster.


On Tue, Aug 13, 2013 at 7:06 PM, Pankaj Gupta <pa...@brightroll.com> wrote:

> Looking at the code it seems like the lock and the i/o is done per event
> and not for a batch. Is that correct? If that is the case then it seems
> like there is a lot of overhead per event. The throughput I'm seeing is 1 -
> 1.5 MBps per disk which is way below the sequential read/write capacity of
> the disk which is easily over 50MBps. Adding more sinks doesn't help, they
> just block waiting for the queue to become free. CPU usage is 20%, there is
> enough RAM for page cache so that no read is going to disk. The queue seems
> to be the bottleneck. What is the throughput I should expect per disk?
>
>
> On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland <br...@cloudera.com> wrote:
>
>> The lock is per file. Adding more directories to the channel will cause
>> more files to be created. Of course you'll need additional disks behind
>> those directories to see any performance increase.
>>
>>
>> On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <
>> hshreedharan@cloudera.com> wrote:
>>
>>>  Yes, IO is done inside locks to avoid multiple takes and puts getting
>>> written out at the same time. Even though Java makes sure the writes are
>>> serialized, Flume still needs to keep track of some counters etc, so the
>>> lock is required. Note that the lock you are talking about is  in the
>>> LogFile class, which represents a single file - so even if the write is
>>> inside that lock (which is also inside that class itself) that  does not
>>> cause any contention - because the lock is just preventing two IO ops to
>>> happen at the same time.
>>>
>>>
>>> Thanks,
>>> Hari
>>>
>>> On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:
>>>
>>> It seems like some i/o is done inside the lock, which means that time
>>> for taking a lock is proportional to the time for i/o and thus it becomes a
>>> problem. I apologize in advance if I am wrong but the call stack and
>>> behavior I'm seeing seems to suggest that. Specifically, it seems that we
>>> do a write while inside take:
>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>>>    java.lang.Thread.State: RUNNABLE
>>>         at sun.nio.ch.NativeThread.current(Native Method)
>>>         at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>>>         at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>>>         - locked <0x00000005190ec998> (a java.lang.Object)
>>>         at
>>> org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>>>         at
>>> org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>>>         - locked <0x0000000519111590> (a
>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>>         at org.apache.flume.channel.file.Log.take(Log.java:550)
>>>         at
>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>>>         at
>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>         at
>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>         at
>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>         at
>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>         at
>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>         at java.lang.Thread.run(Thread.java:662)
>>>
>>>
>>>
>>> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <
>>> hshreedharan@cloudera.com> wrote:
>>>
>>> Since the channel is designed to make sure that events are not
>>> duplicated to multiple sinks, and to protect against corruption due to
>>> concurrency issues, we do not need the locking in the channel's flume event
>>> queue. It is unlikely that locking is what is causing performance issues
>>> because the channel is heavily I/O bound. If you take a series of thread
>>> dumps, you will probably see that those threads are moving forward and the
>>> ones reading/writing from/to disk are the ones which are slower. These
>>> locks are unlikely to hit performance much.
>>>
>>> Thanks,
>>> Hari
>>>
>>> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
>>>
>>> Hi,
>>>
>>> Spent some more time debugging issues with FileChannel. The problem
>>> seems to lock contention reading from FlumeEventQueue:
>>>
>>> I see a lot of threads like this:
>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
>>> [0x00007f821afee000]
>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>         at
>>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>>>         - waiting to lock <0x0000000518ee4c90> (a
>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>         at
>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>>>         at
>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>         at
>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>         at
>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>         at
>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>         at
>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>         at java.lang.Thread.run(Thread.java:662)
>>>
>>>
>>> I have two file channels and 8 Avro Sinks per file channel. I added more
>>> sinks because they weren't draining fast enough. It seems like they send
>>> the batch then wait for an ack before sending again, thus sends are not
>>> pipelined and having more sinks seemed like a good way of getting some
>>> parallelism.
>>>
>>> Here's the full stack trace:
>>> 2013-08-13 15:30:32
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02 mixed
>>> mode):
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000056d932120> (a
>>> java.util.concurrent.CountDownLatch$Sync)
>>>  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>>>  at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>>>  at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
>>> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
>>>  at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
>>> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
>>>  at
>>> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
>>> at $Proxy7.appendBatch(Unknown Source)
>>>  at
>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
>>> at
>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
>>>  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000 nid=0x41a1
>>> waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>  at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005194c3340> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005191a16d8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519174e08> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519174e98> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005195f9670> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519174ce8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005191a1918> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005191e7f60> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153 runnable
>>> [0x00007f8210e4d000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
>>> - locked <0x000000054a2917a8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519174f58> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146 runnable
>>> [0x00007f8210f4e000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000546e5dd28> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118 runnable
>>> [0x00007f821104f000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000544f763c0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111 runnable
>>> [0x00007f8211150000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
>>> - locked <0x000000054473d5a8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110 runnable
>>> [0x00007f8211251000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005437f4f70> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f runnable
>>> [0x00007f8211352000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005422a9018> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e runnable
>>> [0x00007f8211453000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000541063d50> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d runnable
>>> [0x00007f8211554000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000540dc1110> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a runnable
>>> [0x00007f8211655000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000053fbbc3b0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109 runnable
>>> [0x00007f8211756000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
>>> - locked <0x000000053f812860> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd runnable
>>> [0x00007f8211857000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
>>> - locked <0x000000053e6e5570> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519175138> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519174c28> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005191751c8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0 runnable
>>> [0x00007f8211c5b000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000536579db0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005191750a8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1 runnable
>>> [0x00007f8211e5d000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005313943a8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051936f460> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051936dfa8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de runnable
>>> [0x00007f8212160000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
>>> - locked <0x000000052dad4468> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd runnable
>>> [0x00007f8212261000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
>>> - locked <0x000000052c73c5f0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc runnable
>>> [0x00007f8212362000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
>>> - locked <0x000000052b8da998> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db waiting
>>> for monitor entry [0x00007f8212462000]
>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>> at
>>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>>> - waiting to lock <0x0000000518ee4c90> (a
>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>  at
>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>>> at
>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>>  at
>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>> at
>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>>  at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>  at
>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>>  at
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>  at
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>  at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>>  at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>>  at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>  at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>>  at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>  at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051936df18> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000518fe7ae8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8 runnable
>>> [0x00007f8212766000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005263d6ba8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7 runnable
>>> [0x00007f8212867000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000525d1d0e0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6 runnable
>>> [0x00007f8212968000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000525a689d8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5 runnable
>>> [0x00007f8212a69000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000524e1e7d8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4 runnable
>>> [0x00007f8212b6a000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000524d437a8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-36" prio=10 tid=0x00007f85843ad800 nid=0x40d3 runnable
>>> [0x00007f8212c6b000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000524cf4140> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000524cf4128> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000524ab32e8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-35" prio=10 tid=0x00007f85d83a9000 nid=0x40d2 runnable
>>> [0x00007f8212d6c000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005244877c8> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005244877b0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000523f5b660> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-34" prio=10 tid=0x00007f85d838e000 nid=0x40d1 runnable
>>> [0x00007f8212e6d000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000052419e490> (a sun.nio.ch.Util$2)
>>> - locked <0x000000052419e478> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000523f5b6f0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-34" prio=10 tid=0x00007f8588390800 nid=0x40d0 waiting
>>> for monitor entry [0x00007f8212f6d000]
>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
>>> - waiting to lock <0x0000000519111590> (a
>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>>  at org.apache.flume.channel.file.Log.put(Log.java:507)
>>> at
>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
>>>  at
>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>> at
>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>>  at
>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>>  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>  at java.lang.reflect.Method.invoke(Method.java:597)
>>> at
>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>>  at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>> at
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>>  at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>> at
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>>  at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>>  at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>>  at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>>  at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>  at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>>  at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-35" prio=10 tid=0x00007f85843ab800 nid=0x40cf runnable
>>> [0x00007f821306f000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000523d40938> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000523d40aa0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000523ecec18> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-37" prio=10 tid=0x00007f85803e3800 nid=0x40ce waiting
>>> for monitor entry [0x00007f821316f000]
>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>> at
>>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>>> - waiting to lock <0x0000000518ee4c90> (a
>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>  at
>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>>> at
>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>>  at
>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>> at
>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>>  at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>  at
>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>>  at
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>  at
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>  at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>>  at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>>  at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>  at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>>  at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-34" prio=10 tid=0x00007f8584391000 nid=0x40cd runnable
>>> [0x00007f8213271000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> - locked <0x0000000523935540> (a sun.nio.ch.Util$2)
>>>  - locked <0x0000000523935528> (a java.util.Collections$UnmodifiableSet)
>>> - locked <0x0000000523788ed8> (a sun.nio.ch.EPollSelectorImpl)
>>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-33" prio=10 tid=0x00007f85d8373800 nid=0x40cc runnable
>>> [0x00007f8213372000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> - locked <0x00000005239ec658> (a sun.nio.ch.Util$2)
>>>  - locked <0x00000005239ec640> (a java.util.Collections$UnmodifiableSet)
>>> - locked <0x0000000523845410> (a sun.nio.ch.EPollSelectorImpl)
>>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-33" prio=10 tid=0x00007f8588375800 nid=0x40cb waiting
>>> for monitor entry [0x00007f8213472000]
>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
>>>  - waiting to lock <0x0000000519111590> (a
>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>> at org.apache.flume.channel.file.Log.put(Log.java:507)
>>>  at
>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
>>> at
>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>>  at
>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>> at
>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>>  at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>  at
>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>>  at
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>  at
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>  at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>>  at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>>  at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>  at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>>  at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-33" prio=10 tid=0x00007f8584376000 nid=0x40ca runnable
>>> [0x00007f8213574000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> - locked <0x00000005237e3028> (a sun.nio.ch.Util$2)
>>>  - locked <0x00000005237e3010> (a java.util.Collections$UnmodifiableSet)
>>> - locked <0x0000000523788e90> (a sun.nio.ch.EPollSelectorImpl)
>>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-32" prio=10 tid=0x00007f85d8358800 nid=0x40c9 waiting
>>> for monitor entry [0x00007f8213674000]
>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>> at
>>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>>>  - waiting to lock <0x0000000518ee4c90> (a
>>> org.apache.flume.channel.file.FlumeEventQueue)
>>> at
>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>>>  at
>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>> at
>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>>  at
>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>>  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>  at java.lang.reflect.Method.invoke(Method.java:597)
>>> at
>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>>  at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>> at
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>>  at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>> at
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>>  at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>>  at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>>  at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>>  at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>  at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>>  at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-32" prio=10 tid=0x00007f858835b000 nid=0x40c8 runnable
>>> [0x00007f8213776000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000523343688> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000523343670> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000052305e5c8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-36" prio=10 tid=0x00007f85803c9000 nid=0x40c7 runnable
>>> [0x00007f8213877000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005230fa580> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005230fa568> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000522f5ca50> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-32" prio=10 tid=0x00007f858435b000 nid=0x40c6 runnable
>>> [0x00007f8213978000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005231367a8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000523136790> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000052305e658> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-31" prio=10 tid=0x00007f85d833e000 nid=0x40c5 runnable
>>> [0x00007f8213a79000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000522db30a0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000522db3088> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000522bda430> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-31" prio=10 tid=0x00007f8588340000 nid=0x40c4 runnable
>>> [0x00007f8213b7a000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000522c16518> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000522c16500> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000522bd97e8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-30" prio=10 tid=0x00007f85d8323800 nid=0x40c3 runnable
>>> [0x00007f8213c7b000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000522bd1a20> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000522bd1a08> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000522bda3a0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-31" prio=10 tid=0x00007f8584340800 nid=0x40c2 runnable
>>> [0x00007f8213d7c000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005228d1608> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005228d15f0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005228a6aa0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-30" prio=10 tid=0x00007f8588325000 nid=0x40c1 runnable
>>> [0x00007f8213e7d000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000052277b088> (a sun.nio.ch.Util$2)
>>> - locked <0x000000052277b070> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005225ed3b8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-35" prio=10 tid=0x00007f85803ae000 nid=0x40c0 runnable
>>> [0x00007f8213f7e000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000522737380> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000522737368> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005225ed328> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-29" prio=10 tid=0x00007f85d8308800 nid=0x40bf runnable
>>> [0x00007f821407f000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005226034a8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000522603490> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005225ed298> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-28" prio=10 tid=0x00007f85d82ed800 nid=0x40be runnable
>>> [0x00007f8214180000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000522359408> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005223593f0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005221e7278> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-27" prio=10 tid=0x00007f85d82d2800 nid=0x40bd runnable
>>> [0x00007f8214281000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005222c0478> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005222c0460> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005221e6e40> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-29" prio=10 tid=0x00007f858830a800 nid=0x40bc runnable
>>> [0x00007f8214382000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000521ddf488> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000521ddf470> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000521c88d38> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-30" prio=10 tid=0x00007f8584326000 nid=0x40bb waiting
>>> for monitor entry [0x00007f8214482000]
>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
>>> - waiting to lock <0x0000000519111590> (a
>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>>  at org.apache.flume.channel.file.Log.put(Log.java:507)
>>> at
>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
>>>  at
>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>> at
>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>>  at
>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>>  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>  at java.lang.reflect.Method.invoke(Method.java:597)
>>> at
>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>>  at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>> at
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>>  at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>> at
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>>  at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>>  at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>>  at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>>  at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>  at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>>  at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-26" prio=10 tid=0x00007f85d82b8000 nid=0x40ba runnable
>>> [0x00007f8214584000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000521cbcb90> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000521cbcb78> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000521279250> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-28" prio=10 tid=0x00007f85882ef800 nid=0x40b9 runnable
>>> [0x00007f8214685000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000521a1c2d0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000521a1c2b8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005218f1508> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-34" prio=10 tid=0x00007f8580393800 nid=0x40b8 runnable
>>> [0x00007f8214786000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000052181ce58> (a sun.nio.ch.Util$2)
>>> - locked <0x000000052181ce40> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005212791c0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-29" prio=10 tid=0x00007f858430b800 nid=0x40b7 runnable
>>> [0x00007f8214887000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005219175f0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000521746090> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000521278e60> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-27" prio=10 tid=0x00007f85882d4800 nid=0x40b6 runnable
>>> [0x00007f8214988000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000052163e9c8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000052163e9b0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000521279130> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-33" prio=10 tid=0x00007f8580378800 nid=0x40b5 runnable
>>> [0x00007f8214a89000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000052152cb38> (a sun.nio.ch.Util$2)
>>> - locked <0x000000052152cb20> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000521278d40> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-28" prio=10 tid=0x00007f85842f0800 nid=0x40b4 runnable
>>> [0x00007f8214b8a000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005213ba3e0> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005213ba3c8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000521278020> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-27" prio=10 tid=0x00007f85842d5800 nid=0x40b3 runnable
>>> [0x00007f8214c8b000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005213ed660> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005213ed648> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000521278c20> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-26" prio=10 tid=0x00007f85882b9800 nid=0x40b2 runnable
>>> [0x00007f8214d8c000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000052133b7c0> (a sun.nio.ch.Util$2)
>>> - locked <0x000000052133b7a8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005212790a0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-32" prio=10 tid=0x00007f858035d800 nid=0x40af runnable
>>> [0x00007f8214e8d000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000520e53be0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000520e53bc8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000520f3bb50> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-25" prio=10 tid=0x00007f85d829d000 nid=0x40ae runnable
>>> [0x00007f8214f8e000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000520e09a30> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000520e09a18> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000520f3b6d0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-31" prio=10 tid=0x00007f8580342800 nid=0x40ad runnable
>>> [0x00007f821508f000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000520ef70e0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000520ef70c8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000520f3b208> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-26" prio=10 tid=0x00007f85842ba800 nid=0x40ac runnable
>>> [0x00007f8215190000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000520c9dfc0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000520c9dfa8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000520bb3cc0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-25" prio=10 tid=0x00007f85842a0000 nid=0x40ab runnable
>>> [0x00007f8215291000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000520b941a8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000520b94190> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000520bb30c0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-24" prio=10 tid=0x00007f85d8282800 nid=0x40aa runnable
>>> [0x00007f8215392000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000520952288> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000520952270> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000520356888> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-30" prio=10 tid=0x00007f8580328000 nid=0x40a9 runnable
>>> [0x00007f8215493000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000520a2fa20> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000520a2fa08> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000520983da0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-23" prio=10 tid=0x00007f85d8267800 nid=0x40a8 runnable
>>> [0x00007f8215594000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000052095c470> (a sun.nio.ch.Util$2)
>>> - locked <0x000000052095c458> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000520356840> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-29" prio=10 tid=0x00007f858030d000 nid=0x40a7 runnable
>>> [0x00007f8215695000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000052065b178> (a sun.nio.ch.Util$2)
>>> - locked <0x000000052065b160> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000520357320> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-28" prio=10 tid=0x00007f85802f2800 nid=0x40a6 runnable
>>> [0x00007f8215796000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000052043cad0> (a sun.nio.ch.Util$2)
>>> - locked <0x000000052043cab8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005203567b0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-25" prio=10 tid=0x00007f858829f000 nid=0x40a5 runnable
>>> [0x00007f8215897000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000052041a8a8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000052041a890> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000520356690> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-22" prio=10 tid=0x00007f85d824c800 nid=0x40a4 runnable
>>> [0x00007f8215998000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000052036b380> (a sun.nio.ch.Util$2)
>>> - locked <0x000000052036b368> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000520357290> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-24" prio=10 tid=0x00007f8584285800 nid=0x40a3 runnable
>>> [0x00007f8215a99000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000052007ac40> (a sun.nio.ch.Util$2)
>>> - locked <0x000000052007ac28> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051fe41948> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-27" prio=10 tid=0x00007f85802d8000 nid=0x40a2 runnable
>>> [0x00007f8215b9a000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005201a2ae8> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005201a2ad0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051ffda458> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-21" prio=10 tid=0x00007f85d8232000 nid=0x40a1 waiting
>>> for monitor entry [0x00007f8215c9a000]
>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>> at
>>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>>> - waiting to lock <0x0000000518ee4c90> (a
>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>  at
>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>>> at
>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>>  at
>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>> at
>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>>  at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>  at
>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>>  at
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>  at
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>  at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>>  at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>>  at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>  at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>>  at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-26" prio=10 tid=0x00007f85802bd800 nid=0x40a0 runnable
>>> [0x00007f8215d9c000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> - locked <0x000000051ff20e78> (a sun.nio.ch.Util$2)
>>>  - locked <0x000000051ff20e60> (a java.util.Collections$UnmodifiableSet)
>>> - locked <0x000000051fe41900> (a sun.nio.ch.EPollSelectorImpl)
>>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-24" prio=10 tid=0x00007f8588284000 nid=0x409f runnable
>>> [0x00007f8215e9d000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> - locked <0x000000051fbc1ca0> (a sun.nio.ch.Util$2)
>>>  - locked <0x000000051fbc1c88> (a java.util.Collections$UnmodifiableSet)
>>> - locked <0x000000051f9a7978> (a sun.nio.ch.EPollSelectorImpl)
>>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-25" prio=10 tid=0x00007f85802a2800 nid=0x409e runnable
>>> [0x00007f8215f9e000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> - locked <0x000000051fa3b5b8> (a sun.nio.ch.Util$2)
>>>  - locked <0x000000051fa3b5a0> (a java.util.Collections$UnmodifiableSet)
>>> - locked <0x000000051f3155b0> (a sun.nio.ch.EPollSelectorImpl)
>>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-20" prio=10 tid=0x00007f85d8217800 nid=0x409d runnable
>>> [0x00007f821609f000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> - locked <0x000000051f83c838> (a sun.nio.ch.Util$2)
>>>  - locked <0x000000051f83c820> (a java.util.Collections$UnmodifiableSet)
>>> - locked <0x000000051f3150e8> (a sun.nio.ch.EPollSelectorImpl)
>>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-23" prio=10 tid=0x00007f8588269800 nid=0x409c runnable
>>> [0x00007f82161a0000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> - locked <0x000000051f78b9b0> (a sun.nio.ch.Util$2)
>>>  - locked <0x000000051f78b998> (a java.util.Collections$UnmodifiableSet)
>>> - locked <0x000000051f3156d0> (a sun.nio.ch.EPollSelectorImpl)
>>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-22" prio=10 tid=0x00007f858824e800 nid=0x409b runnable
>>> [0x00007f82162a1000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> - locked <0x000000051f7c0cc8> (a sun.nio.ch.Util$2)
>>>  - locked <0x000000051f7c0cb0> (a java.util.Collections$UnmodifiableSet)
>>> - locked <0x000000051f3151c0> (a sun.nio.ch.EPollSelectorImpl)
>>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-23" prio=10 tid=0x00007f858426a800 nid=0x409a runnable
>>> [0x00007f82163a2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> - locked <0x000000051f6065a8> (a sun.nio.ch.Util$2)
>>>  - locked <0x000000051f606590> (a java.util.Collections$UnmodifiableSet)
>>> - locked <0x000000051f3d9668> (a sun.nio.ch.EPollSelectorImpl)
>>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-22" prio=10 tid=0x00007f858424f800 nid=0x4099 runnable
>>> [0x00007f82164a3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> - locked <0x000000051f56d710> (a sun.nio.ch.Util$2)
>>>  - locked <0x000000051f56d6f8> (a java.util.Collections$UnmodifiableSet)
>>> - locked <0x000000051f314f80> (a sun.nio.ch.EPollSelectorImpl)
>>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-21" prio=10 tid=0x00007f8588233800 nid=0x4098 runnable
>>> [0x00007f82165a4000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> - locked <0x000000051f3bcd68> (a sun.nio.ch.Util$2)
>>>  - locked <0x000000051f3bcd50> (a java.util.Collections$UnmodifiableSet)
>>> - locked <0x000000051f315400> (a sun.nio.ch.EPollSelectorImpl)
>>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-19" prio=10 tid=0x00007f85d81fc800 nid=0x4097 runnable
>>> [0x00007f82166a5000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> - locked <0x000000051f53b198> (a sun.nio.ch.Util$2)
>>>  - locked <0x000000051f53b180> (a java.util.Collections$UnmodifiableSet)
>>> - locked <0x000000051f315760> (a sun.nio.ch.EPollSelectorImpl)
>>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-20" prio=10 tid=0x00007f8588218800 nid=0x4096 waiting
>>> for monitor entry [0x00007f82167a5000]
>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
>>>  - waiting to lock <0x0000000519111590> (a
>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>> at org.apache.flume.channel.file.Log.put(Log.java:507)
>>>  at
>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
>>> at
>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>>  at
>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>> at
>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>>  at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>  at
>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>>  at
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>  at
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>  at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>>  at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>>  at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>  at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>>  at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-18" prio=10 tid=0x00007f85d81e1800 nid=0x4095 runnable
>>> [0x00007f82168a7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> - locked <0x000000051ee92ba0> (a sun.nio.ch.Util$2)
>>>  - locked <0x000000051ee92b88> (a java.util.Collections$UnmodifiableSet)
>>> - locked <0x000000051ed19740> (a sun.nio.ch.EPollSelectorImpl)
>>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-19" prio=10 tid=0x00007f85881fd800 nid=0x4094 runnable
>>> [0x00007f82169a8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> - locked <0x000000051ee5da40> (a sun.nio.ch.Util$2)
>>>  - locked <0x000000051ee5da28> (a java.util.Collections$UnmodifiableSet)
>>> - locked <0x000000051e77a940> (a sun.nio.ch.EPollSelectorImpl)
>>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-17" prio=10 tid=0x00007f85d81c6800 nid=0x4093 runnable
>>> [0x00007f8216aa9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> - locked <0x000000051ecb1730> (a sun.nio.ch.Util$2)
>>>  - locked <0x000000051ecb1718> (a java.util.Collections$UnmodifiableSet)
>>> - locked <0x000000051e77a8b0> (a sun.nio.ch.EPollSelectorImpl)
>>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-21" prio=10 tid=0x00007f8584234800 nid=0x4092 runnable
>>> [0x00007f8216baa000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> - locked <0x000000051e899660> (a sun.nio.ch.Util$2)
>>>  - locked <0x000000051e899648> (a java.util.Collections$UnmodifiableSet)
>>> - locked <0x000000051e77a310> (a sun.nio.ch.EPollSelectorImpl)
>>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-24" prio=10 tid=0x00007f8580287000 nid=0x4091 runnable
>>> [0x00007f8216caa000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at
>>> org.apache.flume.channel.file.FlumeEventQueue.addTail(FlumeEventQueue.java:165)
>>>  - locked <0x0000000518ee4c90> (a
>>> org.apache.flume.channel.file.FlumeEventQueue)
>>> at
>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:529)
>>>  - locked <0x0000000518ee4c90> (a
>>> org.apache.flume.channel.file.FlumeEventQueue)
>>> at
>>> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
>>>  at
>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
>>> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>>  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>  at java.lang.reflect.Method.invoke(Method.java:597)
>>> at
>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>>  at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>> at
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>>  at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>> at
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>>  at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>>  at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>>  at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>>  at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>  at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>>  at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-23" prio=10 tid=0x00007f858026c000 nid=0x4090 runnable
>>> [0x00007f8216dac000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051ea8b6f0> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051ea8b6d8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051e77a820> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-16" prio=10 tid=0x00007f85d81ac000 nid=0x408f runnable
>>> [0x00007f8216ead000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051e7c8808> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051e7c87f0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051e77a280> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-20" prio=10 tid=0x00007f8584219800 nid=0x408e runnable
>>> [0x00007f8216fae000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051e8fffa8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051e8fff90> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051e77a700> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-22" prio=10 tid=0x00007f8580251000 nid=0x408d runnable
>>> [0x00007f82170af000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051e4dc398> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051e4dc380> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051dcbeef8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-19" prio=10 tid=0x00007f85841fe800 nid=0x408c runnable
>>> [0x00007f82171b0000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051e221520> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051e221508> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051e407860> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-18" prio=10 tid=0x00007f85841e3800 nid=0x408b runnable
>>> [0x00007f82172b1000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051e465298> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051e465280> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051dcbef88> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-18" prio=10 tid=0x00007f85881e3000 nid=0x408a runnable
>>> [0x00007f82173b2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051dea12e0> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051dea12c8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051cfd4d78> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-15" prio=10 tid=0x00007f85d8191000 nid=0x4089 runnable
>>> [0x00007f82174b3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051de322b8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051de322a0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051dcbec28> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-14" prio=10 tid=0x00007f85d8176800 nid=0x4088 runnable
>>> [0x00007f82175b4000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051dea29a8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051dea2990> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051dcbf018> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-21" prio=10 tid=0x00007f8580236800 nid=0x4087 runnable
>>> [0x00007f82176b5000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051dca95d0> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051dca95b8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051cfd4e08> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-17" prio=10 tid=0x00007f85841c8800 nid=0x4086 runnable
>>> [0x00007f82177b6000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051da8d058> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051da8d040> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051cd693f8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-20" prio=10 tid=0x00007f858021b800 nid=0x4085 runnable
>>> [0x00007f82178b7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051d95a5b0> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051d95a598> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051cfd4e98> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-16" prio=10 tid=0x00007f85841ad800 nid=0x4083 runnable
>>> [0x00007f82179b8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051d92a278> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051d92a260> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051cfd4fb8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-13" prio=10 tid=0x00007f85d815b800 nid=0x4082 runnable
>>> [0x00007f8217ab9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051d734788> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051d734770> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051cd69488> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-19" prio=10 tid=0x00007f8580200800 nid=0x4081 runnable
>>> [0x00007f8217bba000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051d4268e8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051d4268d0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051d139810> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-15" prio=10 tid=0x00007f8584192800 nid=0x4080 runnable
>>> [0x00007f8217cbb000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051d4cda88> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051d4cda70> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051babcf98> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-17" prio=10 tid=0x00007f85881c8000 nid=0x407f runnable
>>> [0x00007f8217dbc000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051d32ffe8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051d32ffd0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051cd691b8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-14" prio=10 tid=0x00007f8584177800 nid=0x407e runnable
>>> [0x00007f8217ebd000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051d506c60> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051d506c48> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051cfd5048> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-16" prio=10 tid=0x00007f85881ad000 nid=0x407d runnable
>>> [0x00007f8217fbe000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051d1223e0> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051d1223c8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051c6c7b40> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-15" prio=10 tid=0x00007f8588192800 nid=0x407c runnable
>>> [0x00007f82180bf000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051d146f50> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051d146f38> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051cd69248> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-13" prio=10 tid=0x00007f858415c800 nid=0x407b runnable
>>> [0x00007f82181c0000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051d11d1b8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051d11d1a0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051c6c7c18> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-12" prio=10 tid=0x00007f85d8141000 nid=0x407a runnable
>>> [0x00007f82182c1000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051cd01a78> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051cd01a60> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051cd695a8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-11" prio=10 tid=0x00007f85d8126800 nid=0x4079 runnable
>>> [0x00007f82183c2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051cfa6f88> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051cfa6f70> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051ba67370> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-14" prio=10 tid=0x00007f8588177800 nid=0x4078 runnable
>>> [0x00007f82184c3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051ce5cdd0> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051ce5cdb8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051c6c7bd0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-12" prio=10 tid=0x00007f8584142000 nid=0x4077 runnable
>>> [0x00007f82185c4000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051cc27cb8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051cc27ca0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051babd0b8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-18" prio=10 tid=0x00007f85801e6000 nid=0x4076 runnable
>>> [0x00007f82186c5000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051cb895c8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051cc77fd0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051c6c7cf0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-11" prio=10 tid=0x00007f8584127800 nid=0x4075 runnable
>>> [0x00007f82187c6000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051ca540d0> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051caad368> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051c6c7e10> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-13" prio=10 tid=0x00007f858815c800 nid=0x4074 runnable
>>> [0x00007f82188c7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051c953ab8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051ca799a8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051babd028> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-17" prio=10 tid=0x00007f85801cb800 nid=0x4073 runnable
>>> [0x00007f82189c8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051c77c3d8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051c8137c8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051c3856f0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-10" prio=10 tid=0x00007f858410c800 nid=0x4072 runnable
>>> [0x00007f8218ac9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051c7dd2b0> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051c613988> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051babcde8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-16" prio=10 tid=0x00007f85801b1000 nid=0x4071 runnable
>>> [0x00007f8218bca000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051c776580> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051c6633c8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051c6c7ea0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-9" prio=10 tid=0x00007f85840f1800 nid=0x4070 runnable
>>> [0x00007f8218ccb000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051c36b550> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051c36b568> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051ba675f8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-8" prio=10 tid=0x00007f85840d6800 nid=0x406f runnable
>>> [0x00007f8218dcc000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051c546b00> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051c40a178> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051ba67760> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-7" prio=10 tid=0x00007f85840bb800 nid=0x406e runnable
>>> [0x00007f8218ecd000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051c2ecf90> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051c29a100> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051ba675b0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-15" prio=10 tid=0x00007f8580196000 nid=0x406d runnable
>>> [0x00007f8218fce000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051c2ee670> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051c2ee688> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051ba67718> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-12" prio=10 tid=0x00007f8588141800 nid=0x406c runnable
>>> [0x00007f82190cf000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051c0bd978> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051c16cc00> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051babcf08> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-6" prio=10 tid=0x00007f85840a0800 nid=0x406b runnable
>>> [0x00007f82191d0000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051c068508> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051bf825f0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051ba676d0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-11" prio=10 tid=0x00007f8588127000 nid=0x406a runnable
>>> [0x00007f82192d1000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051becaa68> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051c03ead0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051babce78> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-10" prio=10 tid=0x00007f858810c800 nid=0x4069 runnable
>>> [0x00007f82193d2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051be31b98> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051bd56790> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051ba67490> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-14" prio=10 tid=0x00007f858017b000 nid=0x4068 runnable
>>> [0x00007f82194d3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051bccb8c0> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051bb14420> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051babcd58> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-5" prio=10 tid=0x00007f8584085800 nid=0x4067 runnable
>>> [0x00007f82195d4000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051bd004a8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051bd49480> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051ba67400> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-4" prio=10 tid=0x00007f858406b000 nid=0x4066 runnable
>>> [0x00007f82196d5000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051ba1e928> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051bccdd00> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051ba66ef0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-9" prio=10 tid=0x00007f85880f1800 nid=0x4065 waiting
>>> for monitor entry [0x00007f82197d5000]
>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
>>> - waiting to lock <0x0000000519111590> (a
>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>>  at org.apache.flume.channel.file.Log.put(Log.java:507)
>>> at
>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
>>>  at
>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>> at
>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>>  at
>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>>  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>  at java.lang.reflect.Method.invoke(Method.java:597)
>>> at
>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>>  at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>> at
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>>  at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>> at
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>>  at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>>  at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>>  at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>>  at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>  at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>>  at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-8" prio=10 tid=0x00007f85880d6800 nid=0x4064 runnable
>>> [0x00007f82198d7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051b5bcca8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051b7ef8f0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051b5c7cb8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-7" prio=10 tid=0x00007f85880bb800 nid=0x4063 runnable
>>> [0x00007f82199d8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051b788ce8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051b788d00> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051b285260> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-13" prio=10 tid=0x00007f8580160800 nid=0x4062 runnable
>>> [0x00007f8219ad9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051b71a0f8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051b750e98> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051aed5a70> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-10" prio=10 tid=0x00007f85d810b800 nid=0x4061 runnable
>>> [0x00007f8219bda000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051b517878> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051b5bab30> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051aed5b90> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-9" prio=10 tid=0x00007f85d80f0800 nid=0x4060 runnable
>>> [0x00007f8219cdb000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051b5656e0> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051b5657a0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051b285338> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-8" prio=10 tid=0x00007f85d80d5800 nid=0x405f runnable
>>> [0x00007f8219ddc000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051b3df220> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051b475118> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051aed5c20> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-6" prio=10 tid=0x00007f85880a0800 nid=0x405e runnable
>>> [0x00007f8219edd000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051b11c010> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051b1eab78> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051af91478> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-7" prio=10 tid=0x00007f85d80bb000 nid=0x405d runnable
>>> [0x00007f8219fde000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051b1d36e0> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051b0328c0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051aed5cb0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-12" prio=10 tid=0x00007f8580145800 nid=0x405c runnable
>>> [0x00007f821a0df000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051aeb3070> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051aeb3088> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051a98a820> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-6" prio=10 tid=0x00007f85d80a0000 nid=0x405b runnable
>>> [0x00007f821a1e0000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051aefa920> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051af3ee40> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051ac28810> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-5" prio=10 tid=0x00007f8588086000 nid=0x405a runnable
>>> [0x00007f821a2e1000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051aefe1b0> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051ad2b858> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051aed5e18> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-11" prio=10 tid=0x00007f858012a800 nid=0x4059 runnable
>>> [0x00007f821a3e2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051ac42588> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051ad05d70> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051a98a7d8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-3" prio=10 tid=0x00007f8584050000 nid=0x4058 runnable
>>> [0x00007f821a4e3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051ad06100> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051ad47428> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051ac289c0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-10" prio=10 tid=0x00007f858010f800 nid=0x4057 runnable
>>> [0x00007f821a5e4000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051aab5d80> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051ab8ce38> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051a98a748> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-5" prio=10 tid=0x00007f85d8085800 nid=0x4056 runnable
>>> [0x00007f821a6e5000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051a962980> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051a775fb0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051a842098> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-2" prio=10 tid=0x00007f8584035000 nid=0x4055 runnable
>>> [0x00007f821a7e6000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051a817210> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051a7d2b48> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051a841ca8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-4" prio=10 tid=0x00007f85d806b000 nid=0x4054 runnable
>>> [0x00007f821a8e7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051a903340> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051a903358> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051a42db50> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-9" prio=10 tid=0x00007f85800f4800 nid=0x4053 runnable
>>> [0x00007f821a9e8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051a4f17d8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051a37ae08> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051a22e340> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-4" prio=10 tid=0x00007f858806b000 nid=0x4052 runnable
>>> [0x00007f821aae9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051a4d81e8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051a616bb0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051a1dfb40> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-3" prio=10 tid=0x00007f85d8050000 nid=0x4051 runnable
>>> [0x00007f821abea000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051a4fb240> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051a3913b8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051a213878> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-2" prio=10 tid=0x00007f85d8035800 nid=0x4050 runnable
>>> [0x00007f821aceb000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051a506248> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051a506308> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051a42dd48> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-8" prio=10 tid=0x00007f85800d9800 nid=0x404f runnable
>>> [0x00007f821adec000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051a2d5170> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051a2d5188> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051a22e2b0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-7" prio=10 tid=0x00007f85800be800 nid=0x404e runnable
>>> [0x00007f821aeed000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051a2d3cb8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051a2b8310> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051a213758> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
>>> [0x00007f821afee000]
>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>> at
>>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>>> - waiting to lock <0x0000000518ee4c90> (a
>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>  at
>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>>> at
>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>  at
>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>> at
>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>  at
>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f836c01a000 nid=0x404c runnable [0x00007f821b0ef000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051a045bb0> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051a045bc8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051a000c98> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b376800
>>> nid=0x404b waiting on condition [0x00007f821b1f0000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051a045c10> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.NativeThread.current(Native Method)
>>> at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>>>  at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>>> - locked <0x00000005190ec998> (a java.lang.Object)
>>>  at org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>>> at org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>>>  - locked <0x0000000519111590> (a
>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>> at org.apache.flume.channel.file.Log.take(Log.java:550)
>>>  at
>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>>> at
>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>  at
>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>> at
>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>  at
>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f8354035000 nid=0x4049 runnable [0x00007f821b3f2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051a045c28> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051a045c40> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051a000ce0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f834801a000 nid=0x4048 runnable [0x00007f821b4f3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051a17bd60> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051a17bd78> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051a180230> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b374800
>>> nid=0x4047 waiting on condition [0x00007f821b5f4000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051a08ba88> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857338a800
>>> nid=0x4046 waiting on condition [0x00007f821b6f5000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051a147940> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>> tid=0x00007f85f3bff800 nid=0x4045 waiting on condition [0x00007f821b7f6000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000056dca65c0> (a
>>> java.util.concurrent.FutureTask$Sync)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
>>>  at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
>>> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:227)
>>>  at java.util.concurrent.FutureTask.get(FutureTask.java:91)
>>> at
>>> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:323)
>>>  at
>>> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:269)
>>> at
>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:352)
>>>  at
>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f8344035000 nid=0x4044 runnable [0x00007f821b8f7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051a184a90> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051a184aa8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051a0bb2e0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bfd800
>>> nid=0x4043 waiting on condition [0x00007f821b9f8000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051a147958> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f833001a000 nid=0x4042 runnable [0x00007f821baf9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051a188f28> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051a188f40> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519d90758> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f8334035000 nid=0x4041 runnable [0x00007f821bbfa000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051a045cd0> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051a045ce8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051a000d28> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b372800
>>> nid=0x4040 waiting on condition [0x00007f821bcfb000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519f74590> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573388800
>>> nid=0x403f waiting on condition [0x00007f821bdfc000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519fb0c90> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f832c035000 nid=0x403e runnable [0x00007f821befd000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051a045d30> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051a045d48> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051a000d70> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bfb800
>>> nid=0x403d waiting on condition [0x00007f821bffe000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519e5d388> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f831801a000 nid=0x403c runnable [0x00007f82341c0000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519ebdb70> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519ebdb88> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519d3a778> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f831001a000 nid=0x403b runnable [0x00007f82342c1000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519f47f90> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519f47fa8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519e11ca8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaf0000
>>> nid=0x403a waiting on condition [0x00007f82343c2000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519f73018> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f8308035000 nid=0x4039 runnable [0x00007f82344c3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519f745a8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519f745c0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519ca9f78> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573386800
>>> nid=0x4038 waiting on condition [0x00007f82345c4000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519fa5ce0> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bf9800
>>> nid=0x4037 waiting on condition [0x00007f82346c5000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519fb0ca8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f8300035000 nid=0x4036 runnable [0x00007f82347c6000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519f22d20> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519f22d38> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519d9f4b8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaee000
>>> nid=0x4035 waiting on condition [0x00007f82348c7000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519fb0cc0> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f82fc01a000 nid=0x4034 runnable [0x00007f82349c8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519f74608> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519f74620> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519ca9fc0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f82f001a000 nid=0x4033 runnable [0x00007f8234ac9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519fb0cd8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519fb0cf0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519cadd40> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f44800
>>> nid=0x4032 waiting on condition [0x00007f8234bca000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519ebdbd0> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3377000
>>> nid=0x4031 waiting on condition [0x00007f8234ccb000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519f701c0> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f82e8035000 nid=0x4030 runnable [0x00007f8234dcc000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519f47ff0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519f48008> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519e11cf0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaec000
>>> nid=0x402f waiting on condition [0x00007f8234ecd000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519fa5cf8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>> tid=0x00007f857fc38000 nid=0x402e waiting for monitor entry
>>> [0x00007f8234fce000]
>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>> at
>>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>>> - waiting to lock <0x0000000518ee4c90> (a
>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>  at
>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>>> at
>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>  at
>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>> at
>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>  at
>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f82dc01a000 nid=0x402d runnable [0x00007f82350cf000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519ebdbe8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519ebdc00> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519d3a7c0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f82d001a000 nid=0x402c runnable [0x00007f82351d0000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519fb0d38> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519fb0d50> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519cadd88> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-1" prio=10 tid=0x00007f85d8001000 nid=0x402b runnable
>>> [0x00007f82352d1000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519f481a0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519f49040> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519e11d80> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6d3800
>>> nid=0x402a waiting on condition [0x00007f82353d2000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519f22d80> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f42800
>>> nid=0x4029 waiting on condition [0x00007f82354d3000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519fa5d10> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f82cc035000 nid=0x4028 runnable [0x00007f82355d4000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519fb0d98> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519fb0db0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519caddd0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-3" prio=10 tid=0x00007f8588001000 nid=0x4027 runnable
>>> [0x00007f82356d5000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519ebdc48> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519ebdc60> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519d3a808> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3375000
>>> nid=0x4026 waiting on condition [0x00007f82357d6000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519fa5d28> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f82c4035000 nid=0x4025 runnable [0x00007f82358d7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519f22d98> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519f22db0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519d9f500> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaea000
>>> nid=0x4024 waiting on condition [0x00007f82359d8000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519fa5d40> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f82a8035000 nid=0x4023 runnable [0x00007f8235ad9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519fb0df8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519fb0e10> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519cade18> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f82b401a000 nid=0x4022 runnable [0x00007f8235bda000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519f73030> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519f73048> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519e69de8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f82ac035000 nid=0x4021 runnable [0x00007f8235cdb000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519f61380> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519f61398> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005197600f0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6d1800
>>> nid=0x4020 waiting on condition [0x00007f8235ddc000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519f22df8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f40800
>>> nid=0x401f waiting on condition [0x00007f8235edd000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519fa5d58> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3373000
>>> nid=0x401e waiting on condition [0x00007f8235fde000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519ebdca8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f82a4035000 nid=0x401d runnable [0x00007f82360df000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519fb0e58> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519fb0e70> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519cade60> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a8000
>>> nid=0x401c waiting on condition [0x00007f82361e0000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519f22e10> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-6" prio=10 tid=0x00007f85800a3800 nid=0x401b runnable
>>> [0x00007f82362e1000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519ebde28> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519de7350> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519d3a898> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f828801a000 nid=0x401a runnable [0x00007f82363e2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519fa5d70> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519fa5d88> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519d36338> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f828c035000 nid=0x4019 runnable [0x00007f82364e3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519f73090> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519f730a8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519e69e30> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f8280035000 nid=0x4018 runnable [0x00007f82365e4000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519fb0eb8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519e1a7f8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519cadea8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6cf800
>>> nid=0x4017 waiting on condition [0x00007f82366e5000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519f22e28> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f3e800
>>> nid=0x4016 waiting on condition [0x00007f82367e6000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519f701d8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3371000
>>> nid=0x4015 waiting on condition [0x00007f82368e7000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519fb0f00> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f8278035000 nid=0x4014 runnable [0x00007f82369e8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519e93628> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519e93640> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519d9f548> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a6000
>>> nid=0x4013 waiting on condition [0x00007f8236ae9000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005199bdaf0> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f827001a000 nid=0x4012 runnable [0x00007f8236bea000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519fb0f18> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519fb0f30> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519cadef0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6cd800
>>> nid=0x4011 waiting on condition [0x00007f8236ceb000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519c47f38> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f826c035000 nid=0x4010 runnable [0x00007f8236dec000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519fa5dd0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519fa5de8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519d36380> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f826001a000 nid=0x400f runnable [0x00007f8236eed000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519c8c170> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519c8c188> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519a73d78> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572aeb000
>>> nid=0x400e waiting on condition [0x00007f8236fee000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519c7e570> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2e800
>>> nid=0x400d waiting on condition [0x00007f82370ef000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519c47f50> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f8258035000 nid=0x400b runnable [0x00007f82371f0000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519c4b190> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519c4b1a8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005199eb068> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a4000
>>> nid=0x400a waiting on condition [0x00007f82372f1000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519c47f68> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f825001a000 nid=0x4009 runnable [0x00007f82373f2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519c33550> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519c33568> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051996b7b8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e29c000
>>> nid=0x4008 waiting on condition [0x00007f82374f3000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519c47f80> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f824c03f800 nid=0x4007 runnable [0x00007f82375f4000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519c7e588> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519c7e5a0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005199a1910> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f8240035000 nid=0x4006 runnable [0x00007f82376f5000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519c47f98> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519c47fb0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005199536b8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae9000
>>> nid=0x4005 waiting on condition [0x00007f82377f6000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519c368c8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2c800
>>> nid=0x4004 waiting on condition [0x00007f82378f7000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519c29bf0> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f8238035000 nid=0x4003 runnable [0x00007f82379f8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519bd8198> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519bd81b0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519b59da0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a2000
>>> nid=0x4002 waiting on condition [0x00007f8237af9000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519c7e5e8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f8230035000 nid=0x4001 runnable [0x00007f8237bfa000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519c47ff8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519c48010> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519953700> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e29a000
>>> nid=0x4000 waiting on condition [0x00007f8237cfb000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519c7e600> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f823001a000 nid=0x3fff runnable [0x00007f8237dfc000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519c7e618> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519c7e630> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005199a1958> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f823c01a000 nid=0x3ffe runnable [0x00007f8237efd000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519c48058> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519c48070> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519a6d798> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f823801a000 nid=0x3ffd runnable [0x00007f8237ffe000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519c335b0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519c335c8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051996b800> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae7000
>>> nid=0x3ffc waiting on condition [0x00007f84001c0000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519c480b8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2a800
>>> nid=0x3ffb waiting on condition [0x00007f84002c1000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519bd81f8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24e800
>>> nid=0x3ffa waiting on condition [0x00007f84003c2000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519c29c08> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f824001a000 nid=0x3ff9 runnable [0x00007f84004c3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519c7e678> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519c7e690> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005199a19a0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e298000
>>> nid=0x3ff8 waiting on condition [0x00007f84005c4000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519c67408> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-5" prio=10 tid=0x00007f8580088800 nid=0x3ff7 runnable
>>> [0x00007f84006c5000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519c48238> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519d5e0d0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519a6d828> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f825c01a000 nid=0x3ff6 runnable [0x00007f84007c6000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519c368e0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519c368f8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519953088> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f825801a000 nid=0x3ff5 runnable [0x00007f84008c7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519c33610> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519c33628> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051996b848> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f826401a000 nid=0x3ff4 runnable [0x00007f84009c8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519c7e6d8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519c7e6f0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005199a19e8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae5000
>>> nid=0x3ff3 waiting on condition [0x00007f8400ac9000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519c730b8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24c800
>>> nid=0x3ff2 waiting on condition [0x00007f8400bca000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519c29c20> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f29000
>>> nid=0x3ff1 waiting on condition [0x00007f8400ccb000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519c4b1f0> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f826c01a000 nid=0x3ff0 runnable [0x00007f8400dcc000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519bd8210> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519bd8228> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519b59de8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e296800
>>> nid=0x3fef waiting on condition [0x00007f8400ecd000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519c8c1d0> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f827801a000 nid=0x3fee runnable [0x00007f8400fce000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519c7e738> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519c7e750> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005199a1a30> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f828401a000 nid=0x3fed runnable [0x00007f84010cf000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519c33670> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519c33688> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051996b890> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f828001a000 nid=0x3fec runnable [0x00007f84011d0000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519c29c38> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519c29c50> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519955c68> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f828c01a000 nid=0x3feb runnable [0x00007f84012d1000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519c7e798> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519c7e7b0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005199533a0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae6000
>>> nid=0x3fea waiting on condition [0x00007f84013d2000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519bd8270> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24a800
>>> nid=0x3fe9 waiting on condition [0x00007f84014d3000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519c36940> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572273800
>>> nid=0x3fe8 waiting on condition [0x00007f84015d4000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519c8c1e8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de54800
>>> nid=0x3fe7 waiting on condition [0x00007f84016d5000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519c67420> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f829001a000 nid=0x3fe6 runnable [0x00007f84017d6000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519c7e7f8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519c7e810> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005199533e8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f82a401a000 nid=0x3fe5 runnable [0x00007f84018d7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519c336d0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519c336e8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051996b8d8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae4000
>>> nid=0x3fe4 waiting on condition [0x00007f84019d8000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519946350> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-4" prio=10 tid=0x00007f858006d800 nid=0x3fe3 runnable
>>> [0x00007f8401ad9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519bd83d8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519f042a0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519b59e78> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f82ac01a000 nid=0x3fe2 runnable [0x00007f8401bda000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519c7e858> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519c7e870> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519953430> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a248800
>>> nid=0x3fe1 waiting on condition [0x00007f8401cdb000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519890ec8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f82a801a000 nid=0x3fe0 runnable [0x00007f8401ddc000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519c8c200> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519c8c218> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519a73dc0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572271800
>>> nid=0x3fdf waiting on condition [0x00007f8401edd000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005198a3170> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de52800
>>> nid=0x3fde waiting on condition [0x00007f8401fde000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005198b7610> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f82b801a000 nid=0x3fdd runnable [0x00007f84020df000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519947f70> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519947f88> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005197bc938> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f82c401a000 nid=0x3fdc runnable [0x00007f84021e0000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519946368> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519946380> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051983b9b8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f82cc01a000 nid=0x3fdb runnable [0x00007f84022e1000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005198a3188> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005198a31a0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005197300e0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae2000
>>> nid=0x3fda waiting on condition [0x00007f84023e2000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005199463c8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857226f800
>>> nid=0x3fd9 waiting on condition [0x00007f84024e3000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519792380> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f82c801a000 nid=0x3fd8 runnable [0x00007f84025e4000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519947fd0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519947fe8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005197bc980> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579597000
>>> nid=0x3fd7 waiting on condition [0x00007f84026e5000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519488cc0> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de50800
>>> nid=0x3fd6 waiting on condition [0x00007f84027e6000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005198cc640> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f82d801a000 nid=0x3fd5 runnable [0x00007f84028e7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005199463e0> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005199463f8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051983ba00> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f82e401a000 nid=0x3fd4 runnable [0x00007f84029e8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005198f7fc8> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005198f7fe0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005194d0160> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f82ec01a000 nid=0x3fd3 runnable [0x00007f8402ae9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005198a31e8> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005198a3200> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519730128> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae0800
>>> nid=0x3fd2 waiting on condition [0x00007f8402bea000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519948030> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857226d800
>>> nid=0x3fd1 waiting on condition [0x00007f8402ceb000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519946440> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f82e801a000 nid=0x3fd0 runnable [0x00007f8402dec000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519890ee0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519890ef8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005196f56f8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579595000
>>> nid=0x3fcf waiting on condition [0x00007f8402eed000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005198b7628> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de4e800
>>> nid=0x3fce waiting on condition [0x00007f8402fee000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005198a3248> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f830401a000 nid=0x3fcd runnable [0x00007f84030ef000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519946458> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519946470> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051983ba48> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f830001a000 nid=0x3fcc runnable [0x00007f84031f0000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519937b88> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519937ba0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519887cb8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f830c01a000 nid=0x3fcb runnable [0x00007f84032f1000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519948048> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519948060> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005197bc9c8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f830801a000 nid=0x3fca runnable [0x00007f84033f2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005199464b8> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005199464d0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051983ba90> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e2b000
>>> nid=0x3fc9 waiting on condition [0x00007f84034f3000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519943980> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1595000
>>> nid=0x3fc8 waiting on condition [0x00007f84035f4000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005198a3260> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857bff5800
>>> nid=0x3fc7 waiting on condition [0x00007f84036f5000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519901138> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da0c000
>>> nid=0x3fc6 waiting on condition [0x00007f84037f6000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005199480a8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f832001a000 nid=0x3fc5 runnable [0x00007f84038f7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519946518> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519946530> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051983bad8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f832c01a000 nid=0x3fc4 runnable [0x00007f84039f8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005198a3278> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005198a3290> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519730170> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f832801a000 nid=0x3fc3 runnable [0x00007f8403af9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005198f8028> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005198f8040> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005194d01a8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f833401a000 nid=0x3fc2 runnable [0x00007f8403bfa000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005198b7640> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005198b7658> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005196f3480> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e29000
>>> nid=0x3fc1 waiting on condition [0x00007f8403cfb000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519946578> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1593000
>>> nid=0x3fc0 waiting on condition [0x00007f8403dfc000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005199480c0> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579592000
>>> nid=0x3fbf waiting on condition [0x00007f8403efd000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519943998> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da0a800
>>> nid=0x3fbe waiting on condition [0x00007f84401c0000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519946590> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f834401a000 nid=0x3fbd runnable [0x00007f84402c1000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519903410> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519903428> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005191a1498> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f834001a000 nid=0x3fbc runnable [0x00007f84403c2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519943ef0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519943f08> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005196f5740> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f834c01a000 nid=0x3fbb runnable [0x00007f84404c3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005199497b0> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005199497c8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519685f60> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573ff1800
>>> nid=0x3fba waiting on condition [0x00007f84405c4000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005198f8088> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1591800
>>> nid=0x3fb9 waiting on condition [0x00007f84406c5000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005198a32d8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579591000
>>> nid=0x3fb8 waiting on condition [0x00007f84407c6000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005199480d8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f835401a000 nid=0x3fb7 runnable [0x00007f84408c7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005199465a8> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005199465c0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051983bb20> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da08800
>>> nid=0x3fb6 waiting on condition [0x00007f84409c8000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005199439b0> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-1" prio=10 tid=0x00007f8584001000 nid=0x3fb5 runnable
>>> [0x00007f8440ac9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005198b77f0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519a00960> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519952e48> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f837401a000 nid=0x3fb4 runnable [0x00007f8440bca000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519937be8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519937c00> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519887d00> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f836801a000 nid=0x3fb3 runnable [0x00007f8440ccb000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519943f50> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519943f68> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005196f5788> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f837001a000 nid=0x3fb2 runnable [0x00007f8440dcc000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519901150> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519901168> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519106100> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f158f800
>>> nid=0x3fb1 waiting on condition [0x00007f8440ecd000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519946608> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e26800
>>> nid=0x3fb0 waiting on condition [0x00007f8440fce000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005198a32f0> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913e000
>>> nid=0x3faf waiting on condition [0x00007f84410cf000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005199439c8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f837801a000 nid=0x3fae runnable [0x00007f84411d0000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005199480f0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519948108> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005197bca10> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da07000
>>> nid=0x3fad waiting on condition [0x00007f84412d1000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519943fb0> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f838801a000 nid=0x3fac runnable [0x00007f84413d2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005198f80a0> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005198f80b8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005194d01f0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f839401a000 nid=0x3fab runnable [0x00007f84414d3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519946620> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519946638> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051983bb68> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f839001a000 nid=0x3faa runnable [0x00007f84415d4000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519937c48> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519937c60> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519887d48> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d0d000
>>> nid=0x3fa9 waiting on condition [0x00007f84416d5000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519654310> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913c000
>>> nid=0x3fa8 waiting on condition [0x00007f84417d6000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005191b54d8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571164000
>>> nid=0x3fa7 waiting on condition [0x00007f84418d7000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519637a20> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f83a801a000 nid=0x3fa6 runnable [0x00007f84419d8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005195f76c8> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005196430c0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005194ecaf0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f83b401a000 nid=0x3fa5 runnable [0x00007f8441ad9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051965f960> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051965f978> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519104648> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f83b001a000 nid=0x3fa4 runnable [0x00007f8441bda000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519655e28> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519655e40> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519104ba0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-3" prio=10 tid=0x00007f8580052800 nid=0x3fa3 runnable
>>> [0x00007f8441cdb000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519637b10> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519694210> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051916e518> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f83bc01a000 nid=0x3fa2 runnable [0x00007f8441ddc000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519597678> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519597690> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519106388> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c4800
>>> nid=0x3fa1 waiting on condition [0x00007f8441edd000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519654328> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d0b000
>>> nid=0x3fa0 waiting on condition [0x00007f8441fde000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519676ad0> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571162000
>>> nid=0x3f9f waiting on condition [0x00007f84420df000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519655618> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913a000
>>> nid=0x3f9e waiting on condition [0x00007f84421e0000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005194ae248> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-2" prio=10 tid=0x00007f8580001000 nid=0x3f9d runnable
>>> [0x00007f84422e1000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519654340> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519654358> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519105380> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f83cc01a000 nid=0x3f9c runnable [0x00007f84423e2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519655e88> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519655ea0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519104be8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f83d001a000 nid=0x3f9b runnable [0x00007f84424e3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005194e1350> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005194e1368> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005191041c8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c3000
>>> nid=0x3f9a waiting on condition [0x00007f84425e4000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051967caf8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f83d801a000 nid=0x3f99 runnable [0x00007f84426e5000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519655ee8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519655f00> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519104c30> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f83dc01a000 nid=0x3f98 runnable [0x00007f84427e6000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005196543a0> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005196543b8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005191053c8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d09800
>>> nid=0x3f97 waiting on condition [0x00007f84428e7000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519655630> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579138000
>>> nid=0x3f96 waiting on condition [0x00007f84429e8000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005196592c8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571160000
>>> nid=0x3f95 waiting on condition [0x00007f8442ae9000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519655f48> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f840801a000 nid=0x3f94 runnable [0x00007f8442bea000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519655648> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519655660> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051914b290> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f83e801a000 nid=0x3f93 runnable [0x00007f8442ceb000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519643108> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519643120> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005194ecb38> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d07800
>>> nid=0x3f92 waiting on condition [0x00007f8442dec000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519658428> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f83f801a000 nid=0x3f91 runnable [0x00007f8442eed000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005194c9908> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005194c9920> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519511968> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f83fc01a000 nid=0x3f90 runnable [0x00007f8442fee000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519643168> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519643180> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005194ecb80> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578465000
>>> nid=0x3f8f waiting on condition [0x00007f8403ffe000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519654400> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857115e800
>>> nid=0x3f8e waiting on condition [0x00007f84430ef000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005196556a8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c1000
>>> nid=0x3f8d waiting on condition [0x00007f84431f0000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519654418> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f840c01a000 nid=0x3f8c runnable [0x00007f84432f1000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005196431c8> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005196431e0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005194ecbc8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f841801a000 nid=0x3f8b runnable [0x00007f84433f2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519655f60> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519655f78> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519104c78> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b4000
>>> nid=0x3f8a waiting on condition [0x00007f84434f3000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005196721e8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5bf000
>>> nid=0x3f89 waiting on condition [0x00007f84435f4000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519654430> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f841c01a000 nid=0x3f88 runnable [0x00007f84436f5000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005196556c0> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005196556d8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051914b2d8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f842801a000 nid=0x3f87 runnable [0x00007f84437f6000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519643228> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519643240> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005194ecc10> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b2000
>>> nid=0x3f86 waiting on condition [0x00007f84438f7000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519654448> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5bd800
>>> nid=0x3f85 waiting on condition [0x00007f84439f8000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051967cb10> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f843001a000 nid=0x3f84 runnable [0x00007f8443af9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519655fc0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519655fd8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519104cc0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b0800
>>> nid=0x3f83 waiting on condition [0x00007f8443bfa000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005196592e0> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f843801a000 nid=0x3f82 runnable [0x00007f8443cfb000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519643288> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005196432a0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005194ecc58> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d17b000
>>> nid=0x3f81 waiting on condition [0x00007f8443dfc000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051921afd8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f843c01a000 nid=0x3f80 runnable [0x00007f8443efd000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519654460> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519654478> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519105410> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08ae800
>>> nid=0x3f7f waiting on condition [0x00007f8443ffe000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519676ae8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f844401a000 nid=0x3f7e runnable [0x00007f85441a6000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051967cb28> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051967cb40> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005194745e0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d179000
>>> nid=0x3f7d waiting on condition [0x00007f85442a7000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005196432e8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f845801a000 nid=0x3f7c runnable [0x00007f85443a8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519658440> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005196583f8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519349128> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f845401a000 nid=0x3f7b runnable [0x00007f85444a9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519676aa0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519676ab8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051921b020> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08ad800
>>> nid=0x3f7a waiting on condition [0x00007f85446c5000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519659298> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578463000
>>> nid=0x3f79 waiting on condition [0x00007f85447c6000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005194a3568> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f845c01a000 nid=0x3f78 runnable [0x00007f85445aa000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005196544c0> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005196544d8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519105458> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d177000
>>> nid=0x3f77 waiting on condition [0x00007f85448c7000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519656020> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f846401a000 nid=0x3f76 runnable [0x00007f85449c8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519655720> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519655738> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519105f98> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857115c800
>>> nid=0x3f75 waiting on condition [0x00007f8544ac9000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519402f58> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f847801a000 nid=0x3f74 runnable [0x00007f8544bca000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519654520> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519654538> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005191054a0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f847401a000 nid=0x3f73 runnable [0x00007f8544ccb000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at java.util.HashMap.newKeyIterator(HashMap.java:840)
>>> at java.util.HashMap$KeySet.iterator(HashMap.java:874)
>>>  at java.util.HashSet.iterator(HashSet.java:153)
>>> at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
>>>  - locked <0x0000000519656248> (a java.util.HashSet)
>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)
>>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>> - locked <0x0000000519656038> (a sun.nio.ch.Util$2)
>>>  - locked <0x0000000519656050> (a java.util.Collections$UnmodifiableSet)
>>> - locked <0x0000000519104d08> (a sun.nio.ch.EPollSelectorImpl)
>>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>> at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0015800
>>> nid=0x3f72 waiting on condition [0x00007f8544dcc000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>  at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051941d010> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578461000
>>> nid=0x3f71 waiting on condition [0x00007f8544ecd000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519477938> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f847c01a000 nid=0x3f70 runnable [0x00007f8544fce000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051942aa08> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051942aa20> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519349050> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d175800
>>> nid=0x3f6f waiting on condition [0x00007f85450cf000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519470c60> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f848401a000 nid=0x3f6e runnable [0x00007f85451d0000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519445fa0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519445fb8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005194679d0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570456800
>>> nid=0x3f6d waiting on condition [0x00007f85452d1000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519419230> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f849801a000 nid=0x3f6c runnable [0x00007f85453d2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519470c78> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519470c90> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005194743a0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f849401a000 nid=0x3f6b runnable [0x00007f85454d3000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051945beb8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051945bed0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519196000> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0013800
>>> nid=0x3f6a waiting on condition [0x00007f85455d4000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519470cd8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857845f000
>>> nid=0x3f69 waiting on condition [0x00007f85456d5000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051941d1c0> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f849c01a000 nid=0x3f68 runnable [0x00007f85457d6000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005194583b8> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005194583d0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005194743e8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c488000
>>> nid=0x3f67 waiting on condition [0x00007f85458d7000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519424e80> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f84a401a000 nid=0x3f66 runnable [0x00007f85459d8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051941d1d8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051941d1f0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519467a18> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570455000
>>> nid=0x3f65 waiting on condition [0x00007f8545ad9000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051945bf18> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f84b801a000 nid=0x3f64 runnable [0x00007f8545bda000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519419248> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519419200> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519431ee0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f84c001a000 nid=0x3f63 runnable [0x00007f8545cdb000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051941d238> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051941d250> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519467a60> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0012000
>>> nid=0x3f62 waiting on condition [0x00007f8545ddc000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519458418> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f84bc01a000 nid=0x3f61 runnable [0x00007f8545edd000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051945bf30> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051945bf48> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519196048> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857845e000
>>> nid=0x3f60 waiting on condition [0x00007f8545fde000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519479f10> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c486000
>>> nid=0x3f5f waiting on condition [0x00007f85460df000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051941d298> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f84c401a000 nid=0x3f5e runnable [0x00007f85461e0000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051942a9a8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051942a9c0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519349098> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570453800
>>> nid=0x3f5d waiting on condition [0x00007f85462e1000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051945bf90> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f84cc01a000 nid=0x3f5c runnable [0x00007f85463e2000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519458430> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519458448> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519474430> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c484800
>>> nid=0x3f5b waiting on condition [0x00007f85464e3000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051941d2b0> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f84d401a000 nid=0x3f5a runnable [0x00007f85465e4000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005194191e8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051936a4e0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519431f28> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570450800
>>> nid=0x3f59 waiting on condition [0x00007f85466e5000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051943ef60> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f84dc01a000 nid=0x3f58 runnable [0x00007f85467e6000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051942a948> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051942a960> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005193490e0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c482800
>>> nid=0x3f57 waiting on condition [0x00007f85468e7000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051945bfa8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f84e401a000 nid=0x3f56 runnable [0x00007f85469e8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051941d2c8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051941d2e0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519467aa8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857044f000
>>> nid=0x3f55 waiting on condition [0x00007f8546ae9000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051945bfc0> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f84f801a000 nid=0x3f54 runnable [0x00007f8546bea000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051936a4c8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051936a480> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519431f70> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f84f401a000 nid=0x3f53 runnable [0x00007f8546ceb000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519458490> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005194584a8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519474478> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578008000
>>> nid=0x3f52 waiting on condition [0x00007f8546dec000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051942a8e8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c00d000
>>> nid=0x3f51 waiting on condition [0x00007f8546eed000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519447098> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f84fc01a000 nid=0x3f50 runnable [0x00007f8546fee000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051941d328> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051941d340> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519467af0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857044d000
>>> nid=0x3f4f waiting on condition [0x00007f85470ef000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051945bfd8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f851001a000 nid=0x3f4e runnable [0x00007f85471f0000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519424e98> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519424e50> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519380d20> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f850c01a000 nid=0x3f4d runnable [0x00007f85472f1000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005194584f0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519458508> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005194744c0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578006800
>>> nid=0x3f4c waiting on condition [0x00007f85473f2000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051945bff0> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c01d800
>>> nid=0x3f4b waiting on condition [0x00007f85474f3000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051936a468> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f851401a000 nid=0x3f4a runnable [0x00007f85475f4000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051941d388> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051941d3a0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519467b38> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857045b800
>>> nid=0x3f49 waiting on condition [0x00007f85476f5000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051942a900> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f853801a000 nid=0x3f48 runnable [0x00007f85477f6000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519479f28> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519479ee0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519203068> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f853001f800 nid=0x3f47 runnable [0x00007f85478f7000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519477950> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519477968> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051922faf0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f852c01f000 nid=0x3f46 runnable [0x00007f85479f8000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005194353f8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519435410> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000051921b0f8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>>> tid=0x00007f853401a000 nid=0x3f45 runnable [0x00007f8547af9000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051945c008> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051945c020> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000519196090> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578005800
>>> nid=0x3f44 waiting on condition [0x00007f8547bfa000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005190de788> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f001a800
>>> nid=0x3f43 waiting on condition [0x00007f8547cfb000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000518fae040> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c022000
>>> nid=0x3f42 waiting on condition [0x00007f8547dfc000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000518cafb20> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570460000
>>> nid=0x3f41 waiting on condition [0x00007f8547efd000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005190db9c8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-2" prio=10 tid=0x00007f8588036000 nid=0x3f40 runnable
>>> [0x00007f8547ffe000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005190ee0e8> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005191f0be8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005190e1d18> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-1" prio=10 tid=0x00007f858801b800 nid=0x3f3f runnable
>>> [0x00007f858c18a000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000518f16658> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051941f820> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005190edc80> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-1" prio=10 tid=0x00007f8580002800 nid=0x3f3e runnable
>>> [0x00007f858c28b000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000051913eb50> (a sun.nio.ch.Util$2)
>>> - locked <0x000000051913eca0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000518c9c4e8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-148-thread-1" prio=10 tid=0x00007f856844c000 nid=0x3f3d waiting on
>>> condition [0x00007f858c38c000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005190cfe98> (a
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-146-thread-1" prio=10 tid=0x00007f85dc006800 nid=0x3f3c waiting on
>>> condition [0x00007f858c48d000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>>  - parking to wait for  <0x0000000519018558> (a
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>  at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-147-thread-1" prio=10 tid=0x00007f85e82b1000 nid=0x3f3b waiting on
>>> condition [0x00007f858c58e000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051903b488> (a
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-145-thread-1" prio=10 tid=0x00007f8574027000 nid=0x3f3a waiting on
>>> condition [0x00007f858c68f000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>>  - parking to wait for  <0x000000051903b998> (a
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>  at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-141-thread-1" prio=10 tid=0x00007f85e8159000 nid=0x3f39 runnable
>>> [0x00007f861cd8e000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005190db9e0> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005190db998> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000518c9c020> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-138-thread-1" prio=10 tid=0x00007f856844a800 nid=0x3f38 runnable
>>> [0x00007f861cf90000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005190e1b68> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005190e1b80> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000518c9c608> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-143-thread-1" prio=10 tid=0x00007f85dc003000 nid=0x3f37 runnable
>>> [0x00007f861d091000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519018fa8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519018fc0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005190e2228> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-137-thread-1" prio=10 tid=0x00007f8574007000 nid=0x3f36 runnable
>>> [0x00007f861ce8f000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000519003dc8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000519003de0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005190ea1a8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "1407844613@qtp-1263282395-1 - Acceptor0 SocketConnector@0.0.0.0:54321"
>>> prio=10 tid=0x00007f85e404a800 nid=0x3f35 runnable [0x00007f858c790000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
>>>  - locked <0x00000005190e2720> (a java.net.SocksSocketImpl)
>>> at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>>>  at java.net.ServerSocket.accept(ServerSocket.java:430)
>>> at org.mortbay.jetty.bio.SocketConnector.accept(SocketConnector.java:99)
>>>  at
>>> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
>>> at
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>
>>> "208924888@qtp-1263282395-0" prio=10 tid=0x00007f85e4021800 nid=0x3f34
>>> in Object.wait() [0x00007f861c15b000]
>>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>  at java.lang.Object.wait(Native Method)
>>>  - waiting on <0x00000005190d0528> (a
>>> org.mortbay.thread.QueuedThreadPool$PoolThread)
>>> at
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
>>>  - locked <0x00000005190d0528> (a
>>> org.mortbay.thread.QueuedThreadPool$PoolThread)
>>>
>>> "lifecycleSupervisor-1-9" prio=10 tid=0x00007f85e401a800 nid=0x3f33
>>> waiting on condition [0x00007f861c687000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000518c85670> (a
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "lifecycleSupervisor-1-8" prio=10 tid=0x00007f85e4018800 nid=0x3f32
>>> waiting on condition [0x00007f861c788000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>>  - parking to wait for  <0x0000000518c85670> (a
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>  at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "lifecycleSupervisor-1-7" prio=10 tid=0x00007f85e4016800 nid=0x3f31
>>> waiting on condition [0x00007f861c889000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000518c85670> (a
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "lifecycleSupervisor-1-6" prio=10 tid=0x00007f85e4014800 nid=0x3f30
>>> waiting on condition [0x00007f861c98a000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>>  - parking to wait for  <0x0000000518c85670> (a
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>  at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "lifecycleSupervisor-1-5" prio=10 tid=0x00007f85e4013000 nid=0x3f2f
>>> waiting on condition [0x00007f861ca8b000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000518c85670> (a
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "lifecycleSupervisor-1-4" prio=10 tid=0x00007f85e4028000 nid=0x3f2e
>>> waiting on condition [0x00007f861cb8c000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>>  - parking to wait for  <0x0000000518c85670> (a
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>  at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "lifecycleSupervisor-1-3" prio=10 tid=0x00007f85e4026800 nid=0x3f2d
>>> waiting on condition [0x00007f861cc8d000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000518c85670> (a
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Log-BackgroundWorker-ch1" prio=10 tid=0x00007f85dc001000 nid=0x3f27
>>> waiting on condition [0x00007f861d299000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>>  - parking to wait for  <0x0000000519017718> (a
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>  at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Log-BackgroundWorker-ch2" prio=10 tid=0x00007f85e80ca000 nid=0x3f26
>>> waiting on condition [0x00007f861d39a000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051903b4e8> (a
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "lifecycleSupervisor-1-2" prio=10 tid=0x00007f85e4025000 nid=0x3f25
>>> waiting on condition [0x00007f861d6ba000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>>  - parking to wait for  <0x0000000518c85670> (a
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>  at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "lifecycleSupervisor-1-1" prio=10 tid=0x00007f85e400e800 nid=0x3f24
>>> waiting on condition [0x00007f861d7bb000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000518c85670> (a
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "conf-file-poller-0" prio=10 tid=0x00007f85f0001000 nid=0x3f23 waiting
>>> on condition [0x00007f861d8bc000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>>  - parking to wait for  <0x0000000518ec72f8> (a
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>  at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "DestroyJavaVM" prio=10 tid=0x00007f86a800c800 nid=0x3efa waiting on
>>> condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>>
>>> "lifecycleSupervisor-1-0" prio=10 tid=0x00007f86a89f5800 nid=0x3f22
>>> waiting on condition [0x00007f861d9bd000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>  at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000518c85670> (a
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>>  at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Low Memory Detector" daemon prio=10 tid=0x00007f86a831d800 nid=0x3f20
>>> runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>>
>>> "C2 CompilerThread1" daemon prio=10 tid=0x00007f86a831b000 nid=0x3f1f
>>> waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>>
>>> "C2 CompilerThread0" daemon prio=10 tid=0x00007f86a8319000 nid=0x3f1e
>>> waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>>
>>> "Signal Dispatcher" daemon prio=10 tid=0x00007f86a8316800 nid=0x3f1d
>>> runnable [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>>
>>> "Surrogate Locker Thread (Concurrent GC)" daemon prio=10
>>> tid=0x00007f86a8314800 nid=0x3f1c waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>>
>>> "Finalizer" daemon prio=10 tid=0x00007f86a82f9800 nid=0x3f1b in
>>> Object.wait() [0x00007f861e3eb000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>>  at java.lang.Object.wait(Native Method)
>>> - waiting on <0x0000000518c843c0> (a java.lang.ref.ReferenceQueue$Lock)
>>>  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>>> - locked <0x0000000518c843c0> (a java.lang.ref.ReferenceQueue$Lock)
>>>  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>>
>>> "Reference Handler" daemon prio=10 tid=0x00007f86a82f7800 nid=0x3f1a in
>>> Object.wait() [0x00007f861e4ec000]
>>>    java.lang.Thread.State: WAITING (on object monitor)
>>> at java.lang.Object.wait(Native Method)
>>>  - waiting on <0x0000000518c80860> (a java.lang.ref.Reference$Lock)
>>> at java.lang.Object.wait(Object.java:485)
>>>  at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>>> - locked <0x0000000518c80860> (a java.lang.ref.Reference$Lock)
>>>
>>> "VM Thread" prio=10 tid=0x00007f86a82f1000 nid=0x3f19 runnable
>>>
>>> "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f86a801b800
>>> nid=0x3efb runnable
>>>
>>> "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f86a801d000
>>> nid=0x3efc runnable
>>>
>>> "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f86a801f000
>>> nid=0x3efd runnable
>>>
>>> "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f86a8021000
>>> nid=0x3efe runnable
>>>
>>> "Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x00007f86a8022800
>>> nid=0x3eff runnable
>>>
>>> "Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x00007f86a8024800
>>> nid=0x3f00 runnable
>>>
>>> "Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x00007f86a8026000
>>> nid=0x3f01 runnable
>>>
>>> "Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x00007f86a8028000
>>> nid=0x3f02 runnable
>>>
>>> "Gang worker#8 (Parallel GC Threads)" prio=10 tid=0x00007f86a802a000
>>> nid=0x3f03 runnable
>>>
>>> "Gang worker#9 (Parallel GC Threads)" prio=10 tid=0x00007f86a802b800
>>> nid=0x3f04 runnable
>>>
>>> "Gang worker#10 (Parallel GC Threads)" prio=10 tid=0x00007f86a802d800
>>> nid=0x3f05 runnable
>>>
>>> "Gang worker#11 (Parallel GC Threads)" prio=10 tid=0x00007f86a802f800
>>> nid=0x3f06 runnable
>>>
>>> "Gang worker#12 (Parallel GC Threads)" prio=10 tid=0x00007f86a8031000
>>> nid=0x3f07 runnable
>>>
>>> "Gang worker#13 (Parallel GC Threads)" prio=10 tid=0x00007f86a8033000
>>> nid=0x3f08 runnable
>>>
>>> "Gang worker#14 (Parallel GC Threads)" prio=10 tid=0x00007f86a8035000
>>> nid=0x3f09 runnable
>>>
>>> "Gang worker#15 (Parallel GC Threads)" prio=10 tid=0x00007f86a8036800
>>> nid=0x3f0a runnable
>>>
>>> "Gang worker#16 (Parallel GC Threads)" prio=10 tid=0x00007f86a8038800
>>> nid=0x3f0b runnable
>>>
>>> "Gang worker#17 (Parallel GC Threads)" prio=10 tid=0x00007f86a803a000
>>> nid=0x3f0c runnable
>>>
>>> "Gang worker#18 (Parallel GC Threads)" prio=10 tid=0x00007f86a803c000
>>> nid=0x3f0d runnable
>>>
>>> "Gang worker#19 (Parallel GC Threads)" prio=10 tid=0x00007f86a803e000
>>> nid=0x3f0e runnable
>>>
>>> "Gang worker#20 (Parallel GC Threads)" prio=10 tid=0x00007f86a803f800
>>> nid=0x3f0f runnable
>>>
>>> "Gang worker#21 (Parallel GC Threads)" prio=10 tid=0x00007f86a8041800
>>> nid=0x3f10 runnable
>>>
>>> "Gang worker#22 (Parallel GC Threads)" prio=10 tid=0x00007f86a8043800
>>> nid=0x3f11 runnable
>>>
>>> "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f86a82aa000
>>> nid=0x3f18 runnable
>>> "Gang worker#0 (Parallel CMS Threads)" prio=10 tid=0x00007f86a829e800
>>> nid=0x3f12 runnable
>>>
>>> "Gang worker#1 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a0000
>>> nid=0x3f13 runnable
>>>
>>> "Gang worker#2 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a2000
>>> nid=0x3f14 runnable
>>>
>>> "Gang worker#3 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a3800
>>> nid=0x3f15 runnable
>>>
>>> "Gang worker#4 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a5800
>>> nid=0x3f16 runnable
>>>
>>> "Gang worker#5 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a7800
>>> nid=0x3f17 runnable
>>>
>>> "VM Periodic Task Thread" prio=10 tid=0x00007f86a8329000 nid=0x3f21
>>> waiting on condition
>>>
>>> JNI global references: 1749
>>>
>>>
>>>
>>> --
>>>
>>>
>>> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 |
>>> pankaj@brightroll.com
>>>
>>> Pankaj Gupta | Software Engineer
>>>
>>> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>>>
>>>
>>> United States | Canada | United Kingdom | Germany
>>>
>>>
>>> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
>>> !
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>>
>>> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 |
>>> pankaj@brightroll.com
>>>
>>> Pankaj Gupta | Software Engineer
>>>
>>> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>>>
>>>
>>> United States | Canada | United Kingdom | Germany
>>>
>>>
>>> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
>>> !
>>>
>>>
>>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>>
>
>
>
> --
>
>
> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com
>
> Pankaj Gupta | Software Engineer
>
> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>
>
> United States | Canada | United Kingdom | Germany
>
>
> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
> !
>



-- 


*P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com

Pankaj Gupta | Software Engineer

*BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com


United States | Canada | United Kingdom | Germany


We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
!

Re: Lock contention in FileChannel

[Hari Shreedharan <hs...@cloudera.com>]

Looks like your channel is able to write a huge number of events but reads
are slow - which does seem like it is an issue with whatever is taking the
events (the channel does more work to write than read - more data is
written to disk and more data is fsycned). One thing I noted only now is
that you are using load balancing sink processors for your sinks - which
means only one sink configured in the sink processor's group is active at
any point in time - no matter how many sinks are in that group. You might
want to increase the number of sink groups or remove the sink groups
altogether, so all sinks can poll the channel at the same time.


On Wed, Aug 14, 2013 at 11:57 AM, Pankaj Gupta <pa...@brightroll.com>wrote:

> I tried increasing the dataDirs to 2 and 4 per disk but doesn't seem to
> help much. I then replaced the avro sink with null sinks and events are
> still filling up in the channel. I tried with both 2 and 4 dataDirs per
> disk and null sink, still don't get throughput higher than 1.5 MBps.
>
>
> On Tue, Aug 13, 2013 at 8:30 PM, Brock Noland <br...@cloudera.com> wrote:
>
>> Increasing the number of file channels will result in more checkpoints.
>> Therefore there will be more io than simply increasing the number of
>> dataDirs.  However, this might be a case where it'd be nice to relax the
>> file channel data consistency constraints a little to get increased
>> throughput. That feature does not exist at present.
>>
>>
>> On Tue, Aug 13, 2013 at 10:16 PM, Pankaj Gupta <pa...@brightroll.com>wrote:
>>
>>> I did try increasing number of FileChannels. At 2 FileChannels per disk
>>> performance seemed to be 25% better. At 4 FileChannels per disk performance
>>> dropped to even below 1 FileChannel per disk. I will try increasing the
>>> dataDirs tomorrow.
>>>
>>>
>>> On Tue, Aug 13, 2013 at 8:06 PM, Brock Noland <br...@cloudera.com>wrote:
>>>
>>>> dataDirs is a comma separated list. Try 3-4 directories and then the
>>>> same test.
>>>> On Aug 13, 2013 9:58 PM, "Pankaj Gupta" <pa...@brightroll.com> wrote:
>>>>
>>>>> Both disks were at around 15-25%.
>>>>>
>>>>>
>>>>> On Tue, Aug 13, 2013 at 7:54 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>
>>>>>> Gotcha. When you run tge test what is tye disk utilization
>>>>>> percentage? Iostat can be used for this.
>>>>>> On Aug 13, 2013 9:47 PM, "Pankaj Gupta" <pa...@brightroll.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Those are the boxes we want to collect data from. They run flume and
>>>>>>> send data through their avro sinks to the avro source on this box. We are
>>>>>>> getting data at a pretty good rate and the problem is in fact that the
>>>>>>> events don't drain from the FileChannel fast enough and the channel fill
>>>>>>> percentage keeps getting higher.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 13, 2013 at 7:41 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>>>
>>>>>>>> What is sending the events to the avro source?
>>>>>>>> On Aug 13, 2013 9:34 PM, "Pankaj Gupta" <pa...@brightroll.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Here's the config:
>>>>>>>>> # define channels, one for each disk
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.channels.ch1.type = FILE
>>>>>>>>> agent1.channels.ch1.checkpointDir = /flume1/checkpoint
>>>>>>>>> agent1.channels.ch1.dataDirs = /flume1/data
>>>>>>>>> agent1.channels.ch1.maxFileSize = 375809638400
>>>>>>>>> agent1.channels.ch1.capacity = 75000000
>>>>>>>>> agent1.channels.ch1.transactionCapacity = 4000
>>>>>>>>>
>>>>>>>>> agent1.channels.ch2.type = FILE
>>>>>>>>> agent1.channels.ch2.checkpointDir = /flume2/checkpoint
>>>>>>>>> agent1.channels.ch2.dataDirs = /flume2/data
>>>>>>>>> agent1.channels.ch2.maxFileSize = 375809638400
>>>>>>>>> agent1.channels.ch2.capacity = 75000000
>>>>>>>>> agent1.channels.ch2.transactionCapacity = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # Define an Avro source named avroSource1
>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sources.avroSource1.channels = ch1
>>>>>>>>> agent1.sources.avroSource1.type = avro
>>>>>>>>> agent1.sources.avroSource1.bind = 0.0.0.0
>>>>>>>>> agent1.sources.avroSource1.port = <port>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink1-1-1.type = avro
>>>>>>>>> agent1.sinks.avroSink1-1-1.channel = ch1
>>>>>>>>> agent1.sinks.avroSink1-1-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink1-1-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink1-1-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink1-1-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink1-2-1.type = avro
>>>>>>>>> agent1.sinks.avroSink1-2-1.channel = ch1
>>>>>>>>> agent1.sinks.avroSink1-2-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink1-2-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink1-2-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink1-2-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink1-3-1.type = avro
>>>>>>>>> agent1.sinks.avroSink1-3-1.channel = ch1
>>>>>>>>> agent1.sinks.avroSink1-3-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink1-3-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink1-3-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink1-3-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink1-4-1.type = avro
>>>>>>>>> agent1.sinks.avroSink1-4-1.channel = ch1
>>>>>>>>> agent1.sinks.avroSink1-4-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink1-4-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink1-4-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink1-4-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>> which round robin between different hops
>>>>>>>>> agent1.sinkgroups.group1.sinks = avroSink1-1-1 avroSink1-2-1
>>>>>>>>> avroSink1-3-1 avroSink1-4-1
>>>>>>>>> agent1.sinkgroups.group1.processor.type = load_balance
>>>>>>>>> agent1.sinkgroups.group1.processor.selector = ROUND_ROBIN
>>>>>>>>> agent1.sinkgroups.group1.processor.backoff = true
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> #End of set
>>>>>>>>>
>>>>>>>>> # Define an Avro source named avroSource2
>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sources.avroSource2.channels = ch2
>>>>>>>>> agent1.sources.avroSource2.type = avro
>>>>>>>>> agent1.sources.avroSource2.bind = 0.0.0.0
>>>>>>>>> agent1.sources.avroSource2.port = <port>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink2-1-1.type = avro
>>>>>>>>> agent1.sinks.avroSink2-1-1.channel = ch2
>>>>>>>>> agent1.sinks.avroSink2-1-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink2-1-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink2-1-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink2-1-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink2-2-1.type = avro
>>>>>>>>> agent1.sinks.avroSink2-2-1.channel = ch2
>>>>>>>>> agent1.sinks.avroSink2-2-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink2-2-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink2-2-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink2-2-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink2-3-1.type = avro
>>>>>>>>> agent1.sinks.avroSink2-3-1.channel = ch2
>>>>>>>>> agent1.sinks.avroSink2-3-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink2-3-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink2-3-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink2-3-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink2-4-1.type = avro
>>>>>>>>> agent1.sinks.avroSink2-4-1.channel = ch2
>>>>>>>>> agent1.sinks.avroSink2-4-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink2-4-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink2-4-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink2-4-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>> which round robin between different hops
>>>>>>>>> agent1.sinkgroups.group2.sinks = avroSink2-1-1 avroSink2-2-1
>>>>>>>>> avroSink2-3-1 avroSink2-4-1
>>>>>>>>> agent1.sinkgroups.group2.processor.type = load_balance
>>>>>>>>> agent1.sinkgroups.group2.processor.selector = ROUND_ROBIN
>>>>>>>>> agent1.sinkgroups.group2.processor.backoff = true
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> #End of set
>>>>>>>>>
>>>>>>>>> # Define an Avro source named avroSource3
>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sources.avroSource3.channels = ch1
>>>>>>>>> agent1.sources.avroSource3.type = avro
>>>>>>>>> agent1.sources.avroSource3.bind = 0.0.0.0
>>>>>>>>> agent1.sources.avroSource3.port = <port>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink3-1-1.type = avro
>>>>>>>>> agent1.sinks.avroSink3-1-1.channel = ch1
>>>>>>>>> agent1.sinks.avroSink3-1-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink3-1-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink3-1-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink3-1-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink3-2-1.type = avro
>>>>>>>>> agent1.sinks.avroSink3-2-1.channel = ch1
>>>>>>>>> agent1.sinks.avroSink3-2-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink3-2-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink3-2-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink3-2-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink3-3-1.type = avro
>>>>>>>>> agent1.sinks.avroSink3-3-1.channel = ch1
>>>>>>>>> agent1.sinks.avroSink3-3-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink3-3-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink3-3-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink3-3-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink3-4-1.type = avro
>>>>>>>>> agent1.sinks.avroSink3-4-1.channel = ch1
>>>>>>>>> agent1.sinks.avroSink3-4-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink3-4-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink3-4-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink3-4-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>> which round robin between different hops
>>>>>>>>> agent1.sinkgroups.group3.sinks = avroSink3-1-1 avroSink3-2-1
>>>>>>>>> avroSink3-3-1 avroSink3-4-1
>>>>>>>>> agent1.sinkgroups.group3.processor.type = load_balance
>>>>>>>>> agent1.sinkgroups.group3.processor.selector = ROUND_ROBIN
>>>>>>>>> agent1.sinkgroups.group3.processor.backoff = true
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> #End of set
>>>>>>>>>
>>>>>>>>> # Define an Avro source named avroSource4
>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sources.avroSource4.channels = ch2
>>>>>>>>> agent1.sources.avroSource4.type = avro
>>>>>>>>> agent1.sources.avroSource4.bind = 0.0.0.0
>>>>>>>>> agent1.sources.avroSource4.port = <port>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink4-1-1.type = avro
>>>>>>>>> agent1.sinks.avroSink4-1-1.channel = ch2
>>>>>>>>> agent1.sinks.avroSink4-1-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink4-1-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink4-1-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink4-1-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink4-2-1.type = avro
>>>>>>>>> agent1.sinks.avroSink4-2-1.channel = ch2
>>>>>>>>> agent1.sinks.avroSink4-2-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink4-2-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink4-2-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink4-2-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink4-3-1.type = avro
>>>>>>>>> agent1.sinks.avroSink4-3-1.channel = ch2
>>>>>>>>> agent1.sinks.avroSink4-3-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink4-3-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink4-3-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink4-3-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink4-4-1.type = avro
>>>>>>>>> agent1.sinks.avroSink4-4-1.channel = ch2
>>>>>>>>> agent1.sinks.avroSink4-4-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink4-4-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink4-4-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink4-4-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>> which round robin between different hops
>>>>>>>>> agent1.sinkgroups.group4.sinks = avroSink4-1-1 avroSink4-2-1
>>>>>>>>> avroSink4-3-1 avroSink4-4-1
>>>>>>>>> agent1.sinkgroups.group4.processor.type = load_balance
>>>>>>>>> agent1.sinkgroups.group4.processor.selector = ROUND_ROBIN
>>>>>>>>> agent1.sinkgroups.group4.processor.backoff = true
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> #End of set
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # Finally, now that we've defined all of our components, tell
>>>>>>>>> # agent1 which ones we want to activate.
>>>>>>>>> agent1.channels = ch1 ch2
>>>>>>>>> agent1.sources = avroSource1 avroSource2 avroSource3 avroSource4
>>>>>>>>> agent1.sinks = avroSink1-1-1 avroSink1-2-1 avroSink1-3-1
>>>>>>>>> avroSink1-4-1 avroSink2-1-1 avroSink2-2-1 avroSink2-3-1 avroSink2-4-1
>>>>>>>>> avroSink3-1-1 avroSink3-2-1 avroSink3-3-1 avroSink3-4-1 avroSink4-1-1
>>>>>>>>> avroSink4-2-1 avroSink4-3-1 avroSink4-4-1
>>>>>>>>> agent1.sinkgroups = group1 group2 group3 group4
>>>>>>>>>
>>>>>>>>> There are two disks on the box /flume1 and /flume2
>>>>>>>>> 32 cores of CPU, 32 GB RAM, 1GbPS ethernet.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Aug 13, 2013 at 7:22 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>>>>>
>>>>>>>>>> Can you share your conf file?
>>>>>>>>>> On Aug 13, 2013 9:19 PM, "Hari Shreedharan" <
>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>
>>>>>>>>>>>  Even though the writes are done per batch, they don't go to
>>>>>>>>>>> disk rightaway - commits are the only ones which actually cause an fsync -
>>>>>>>>>>> which is when writes actually go to disk.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Hari
>>>>>>>>>>>
>>>>>>>>>>> On Tuesday, August 13, 2013 at 7:06 PM, Pankaj Gupta wrote:
>>>>>>>>>>>
>>>>>>>>>>> Looking at the code it seems like the lock and the i/o is done
>>>>>>>>>>> per event and not for a batch. Is that correct? If that is the case then it
>>>>>>>>>>> seems like there is a lot of overhead per event. The throughput I'm seeing
>>>>>>>>>>> is 1 - 1.5 MBps per disk which is way below the sequential read/write
>>>>>>>>>>> capacity of the disk which is easily over 50MBps. Adding more sinks doesn't
>>>>>>>>>>> help, they just block waiting for the queue to become free. CPU usage is
>>>>>>>>>>> 20%, there is enough RAM for page cache so that no read is going to disk.
>>>>>>>>>>> The queue seems to be the bottleneck. What is the throughput I should
>>>>>>>>>>> expect per disk?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland <
>>>>>>>>>>> brock@cloudera.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> The lock is per file. Adding more directories to the channel
>>>>>>>>>>> will cause more files to be created. Of course you'll need additional disks
>>>>>>>>>>> behind those directories to see any performance increase.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <
>>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>  Yes, IO is done inside locks to avoid multiple takes and puts
>>>>>>>>>>> getting written out at the same time. Even though Java makes sure the
>>>>>>>>>>> writes are serialized, Flume still needs to keep track of some counters
>>>>>>>>>>> etc, so the lock is required. Note that the lock you are talking about is
>>>>>>>>>>>  in the LogFile class, which represents a single file - so even if the
>>>>>>>>>>> write is inside that lock (which is also inside that class itself) that
>>>>>>>>>>>  does not cause any contention - because the lock is just preventing two IO
>>>>>>>>>>> ops to happen at the same time.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Hari
>>>>>>>>>>>
>>>>>>>>>>> On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:
>>>>>>>>>>>
>>>>>>>>>>> It seems like some i/o is done inside the lock, which means that
>>>>>>>>>>> time for taking a lock is proportional to the time for i/o and thus it
>>>>>>>>>>> becomes a problem. I apologize in advance if I am wrong but the call stack
>>>>>>>>>>> and behavior I'm seeing seems to suggest that. Specifically, it seems that
>>>>>>>>>>> we do a write while inside take:
>>>>>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>>>>>>> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>         at sun.nio.ch.NativeThread.current(Native Method)
>>>>>>>>>>>         at
>>>>>>>>>>> sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>>>>>>>>>>>         at
>>>>>>>>>>> sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>>>>>>>>>>>         - locked <0x00000005190ec998> (a java.lang.Object)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>>>>>>>>>>>         - locked <0x0000000519111590> (a
>>>>>>>>>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>>>>>>>>>>         at org.apache.flume.channel.file.Log.take(Log.java:550)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <
>>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Since the channel is designed to make sure that events are not
>>>>>>>>>>> duplicated to multiple sinks, and to protect against corruption due to
>>>>>>>>>>> concurrency issues, we do not need the locking in the channel's flume event
>>>>>>>>>>> queue. It is unlikely that locking is what is causing performance issues
>>>>>>>>>>> because the channel is heavily I/O bound. If you take a series of thread
>>>>>>>>>>> dumps, you will probably see that those threads are moving forward and the
>>>>>>>>>>> ones reading/writing from/to disk are the ones which are slower. These
>>>>>>>>>>> locks are unlikely to hit performance much.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Hari
>>>>>>>>>>>
>>>>>>>>>>> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Spent some more time debugging issues with FileChannel. The
>>>>>>>>>>> problem seems to lock contention reading from FlumeEventQueue:
>>>>>>>>>>>
>>>>>>>>>>> I see a lot of threads like this:
>>>>>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>>>>>>> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
>>>>>>>>>>> [0x00007f821afee000]
>>>>>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>>>>>>>>>>>         - waiting to lock <0x0000000518ee4c90> (a
>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I have two file channels and 8 Avro Sinks per file channel. I
>>>>>>>>>>> added more sinks because they weren't draining fast enough. It seems like
>>>>>>>>>>> they send the batch then wait for an ack before sending again, thus sends
>>>>>>>>>>> are not pipelined and having more sinks seemed like a good way of getting
>>>>>>>>>>> some parallelism.
>>>>>>>>>>>
>>>>>>>>>>> Here's the full stack trace:
>>>>>>>>>>> 2013-08-13 15:30:32
>>>>>>>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02
>>>>>>>>>>> mixed mode):
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
>>>>>>>>>>>    java.lang.Thread.State: WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x000000056d932120> (a
>>>>>>>>>>> java.util.concurrent.CountDownLatch$Sync)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
>>>>>>>>>>> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
>>>>>>>>>>>  at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
>>>>>>>>>>> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
>>>>>>>>>>>  at
>>>>>>>>>>> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
>>>>>>>>>>> at $Proxy7.appendBatch(Unknown Source)
>>>>>>>>>>>  at
>>>>>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000
>>>>>>>>>>> nid=0x41a1 waiting on condition [0x0000000000000000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x00000005194c3340> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x00000005191a16d8> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x0000000519174e08> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x0000000519174e98> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x00000005195f9670> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x0000000519174ce8> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x00000005191a1918> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x00000005191e7f60> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153
>>>>>>>>>>> runnable [0x00007f8210e4d000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x000000054a2917a8> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x0000000519174f58> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146
>>>>>>>>>>> runnable [0x00007f8210f4e000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x0000000546e5dd28> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118
>>>>>>>>>>> runnable [0x00007f821104f000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x0000000544f763c0> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111
>>>>>>>>>>> runnable [0x00007f8211150000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x000000054473d5a8> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110
>>>>>>>>>>> runnable [0x00007f8211251000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x00000005437f4f70> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f
>>>>>>>>>>> runnable [0x00007f8211352000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x00000005422a9018> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e
>>>>>>>>>>> runnable [0x00007f8211453000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x0000000541063d50> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d
>>>>>>>>>>> runnable [0x00007f8211554000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x0000000540dc1110> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a
>>>>>>>>>>> runnable [0x00007f8211655000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x000000053fbbc3b0> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109
>>>>>>>>>>> runnable [0x00007f8211756000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x000000053f812860> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd
>>>>>>>>>>> runnable [0x00007f8211857000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x000000053e6e5570> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x0000000519175138> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x0000000519174c28> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x00000005191751c8> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0
>>>>>>>>>>> runnable [0x00007f8211c5b000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x0000000536579db0> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x00000005191750a8> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1
>>>>>>>>>>> runnable [0x00007f8211e5d000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x00000005313943a8> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x000000051936f460> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x000000051936dfa8> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de
>>>>>>>>>>> runnable [0x00007f8212160000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x000000052dad4468> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd
>>>>>>>>>>> runnable [0x00007f8212261000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x000000052c73c5f0> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc
>>>>>>>>>>> runnable [0x00007f8212362000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x000000052b8da998> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db
>>>>>>>>>>> waiting for monitor entry [0x00007f8212462000]
>>>>>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>>>>>>>>>>> - waiting to lock <0x0000000518ee4c90> (a
>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>>>>>  at
>>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>>>>>>>>>>  at
>>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>>>>>>>>>>  at
>>>>>>>>>>> org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>>>>>>>>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>>>>  at
>>>>>>>>>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>>>>>>>>>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>>>>>>>>>>  at
>>>>>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>>>>>  at
>>>>>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x000000051936df18> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x0000000518fe7ae8> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8
>>>>>>>>>>> runnable [0x00007f8212766000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x00000005263d6ba8> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7
>>>>>>>>>>> runnable [0x00007f8212867000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x0000000525d1d0e0> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6
>>>>>>>>>>> runnable [0x00007f8212968000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x0000000525a689d8> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5
>>>>>>>>>>> runnable [0x00007f8212a69000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x0000000524e1e7d8> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4
>>>>>>>>>>> runnable [0x00007f8212b6a000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x0000000524d437a8> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>
>>>>>>>>>>> ...
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>>
>>> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 |
>>> pankaj@brightroll.com
>>>
>>> Pankaj Gupta | Software Engineer
>>>
>>> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>>>
>>>
>>> United States | Canada | United Kingdom | Germany
>>>
>>>
>>> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
>>> !
>>>
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>>
>
>
>
> --
>
>
> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com
>
> Pankaj Gupta | Software Engineer
>
> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>
>
> United States | Canada | United Kingdom | Germany
>
>
> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
> !
>

Re: Lock contention in FileChannel

[Pankaj Gupta <pa...@brightroll.com>]

Just to provide an update here. With sink groups removed and using two
dataDirs per disk, total of 32 avro sinks, I am able to get reasonable
throughput of up to 100 Mbps. @brock thanks for your suggestion about using
multiple dataDirs, even on the same disk, per channel. Seems that we were
not i/o bound but concurrency bound. To get concrete data I added some
logging to AvroRpcSink:
diff --git
a/flume-ng-core/src/main/java/org/apache/flume/sink/AbstractRpcSink.java
b/flume-ng-core/src/main/java/org/apache/flume/sink/AbstractRpcSink.java
index 892c949..68c43ef 100644
--- a/flume-ng-core/src/main/java/org/apache/flume/sink/AbstractRpcSink.java
+++ b/flume-ng-core/src/main/java/org/apache/flume/sink/AbstractRpcSink.java
@@ -315,19 +315,24 @@ public abstract class AbstractRpcSink extends
AbstractSink

   @Override
   public Status process() throws EventDeliveryException {
+    long startTime = System.currentTimeMillis();
     Status status = Status.READY;
     Channel channel = getChannel();
     Transaction transaction = channel.getTransaction();
-
     try {
+      // Measure time spent in this block
       transaction.begin();

       verifyConnection();

       List<Event> batch = Lists.newLinkedList();
+      long takeTimeSum = 0;

       for (int i = 0; i < client.getBatchSize(); i++) {
+
+        long takeStart = System.currentTimeMillis();
         Event event = channel.take();
+        takeTimeSum += (System.currentTimeMillis() - takeStart);

         if (event == null) {
           break;
@@ -349,10 +354,26 @@ public abstract class AbstractRpcSink extends
AbstractSink
           sinkCounter.incrementBatchCompleteCount();
         }
         sinkCounter.addToEventDrainAttemptCount(size);
+
+        long appendBatchStart = System.currentTimeMillis();
         client.appendBatch(batch);
+        long appendBatchTime = System.currentTimeMillis() -
appendBatchStart;
+        if (size > 0) {
+          logger.debug("<append> time for batch-size: " + size + ": " +
appendBatchTime + "ms");
+          logger.debug("<append> rate: " + (appendBatchTime * 1000)/ size);
+        }
       }

       transaction.commit();
+
+      if (size > 0) {
+        long processTime = System.currentTimeMillis() - startTime;
+        logger.debug("<process> time for batch-size: " + size + ": " +
processTime + "ms");
+        logger.debug("<process> rate: " + (processTime * 1000)/ size);
+        logger.debug("<takes> time for batch-size: " + size + ": " +
takeTimeSum + "ms");
+        logger.debug("<take> rate: " + (takeTimeSum * 1000)/ size);
+      }
+
       sinkCounter.addToEventDrainSuccessCount(size);

     } catch (Throwable t) {

Here are some numbers for the time taken per 1000 takes:
<take> rate: 842
<take> rate: 800
<take> rate: 540
<take> rate: 642
<take> rate: 624
<take> rate: 627
<take> rate: 1011
<take> rate: 1632
<take> rate: 1097
<take> rate: 1542
<take> rate: 1650
<take> rate: 375
<take> rate: 365
<take> rate: 628
<take> rate: 127
<take> rate: 1833
<take> rate: 616
<take> rate: 2169
<take> rate: 1120
<take> rate: 625
<take> rate: 529
<take> rate: 152
<take> rate: 649
<take> rate: 137
<take> rate: 147
<take> rate: 138
<take> rate: 67
<take> rate: 820
<take> rate: 765
<take> rate: 920
<take> rate: 913
<take> rate: 410
<take> rate: 700
<take> rate: 500
<take> rate: 41
<take> rate: 644
<take> rate: 383
<take> rate: 1025
<take> rate: 1366

Here are corresponding number for time taken(ms) per 1000 appends
<append> rate: 171
<append> rate: 276
<append> rate: 491
<append> rate: 271
<append> rate: 327
<append> rate: 100
<append> rate: 117
<append> rate: 153
<append> rate: 211
<append> rate: 452
<append> rate: 433
<append> rate: 392
<append> rate: 482
<append> rate: 557
<append> rate: 500
<append> rate: 179
<append> rate: 192
<append> rate: 220
<append> rate: 174
<append> rate: 175
<append> rate: 171
<append> rate: 171
<append> rate: 268
<append> rate: 55
<append> rate: 325
<append> rate: 326
<append> rate: 56
<append> rate: 833
<append> rate: 1666
<append> rate: 142
<append> rate: 192
<append> rate: 172
<append> rate: 165
<append> rate: 130
<append> rate: 283
<append> rate: 68
<append> rate: 385
<append> rate: 210
<append> rate: 549
<append> rate: 317

(sorry about using the word rate here, it's actually the opposite: time per
1000 actions, smaller is better, I realized it a little late.)

With this logging in place I saw numbers of ~1000ms per 1000 takes vs
~300ms per 1000 appends. Please note that append sends events over the wire
and waits for an ok. This being faster than takes clearly indicated
problems with takes. By increasing the dataDirs to 2, both on the same
disk, the take numbers dropped by half and more. Here are the corresponding
take numbers by using two dataDirs per disk instead of one:
<take> rate: 51
<take> rate: 36
<take> rate: 27
<take> rate: 53
<take> rate: 44
<take> rate: 80
<take> rate: 272
<take> rate: 62
<take> rate: 91
<take> rate: 75
<take> rate: 118
<take> rate: 87
<take> rate: 31
<take> rate: 46
<take> rate: 72
<take> rate: 260
<take> rate: 339
<take> rate: 332
<take> rate: 280
<take> rate: 100
<take> rate: 113
<take> rate: 63
<take> rate: 277
<take> rate: 364
<take> rate: 278
<take> rate: 120
<take> rate: 214
<take> rate: 290
<take> rate: 39
<take> rate: 76
<take> rate: 61
<take> rate: 96
<take> rate: 129
<take> rate: 153
<take> rate: 85
<take> rate: 61
<take> rate: 47
<take> rate: 141
<take> rate: 350
<take> rate: 308
<take> rate: 157

Please also note that the takes above don't even involve a commit, so no
flush to disk. I strongly suspect that the bottleneck here is not just disk
i/o but also locks. I also saw in code that both put and take in logFile
use the same lock. So the contention is not just between sinks but source
and the sink. I don't know the details of the design so not sure why there
is a requirement of taking the same lock for reads and writes but if they
could be different then that should help with performance.

100Mbps is much better than 25Mbps, that I was getting earlier, but it is
way lower than sequential disk i/o throughput of two disks which should be
100MBps i.e. 800Mbps. With two dataDirs per disk %util is already ~40% so I
don't want to take it further to 3 dataDirs per disk. 100Mbps is sufficient
for my use case right now though.

I hope you find the above numbers useful. If you could tell me about the
downsides of not using any sink groups that would be great, but to keep it
clean I'll ask that as a separate question on this list. Thanks for all
your help so far.


On Wed, Aug 14, 2013 at 11:04 PM, Pankaj Gupta <pa...@brightroll.com>wrote:

> I am getting very good performance by removing the groups altogether and
> keeping a high number of total sinks (64). If I organize sinks into groups
> of 4 I loose parallelism and don't get as good performance as with 64
> sinks. The reason I want to organize sinks into groups is for failover. Do
> I get any failover if I don't use any groups? E.g. if I have 64 sinks and
> one of them fails to send events, would some other sink be able to send
> those events or would the failed sink keep trying forever.
>
> The reason I am not able to increase the number of sinks beyond 64 is that
> I start getting failed connection exceptions. It seem like the avro source
> on the destination is not allowing connections beyond a certain number. It
> seems that avro sink creates a thread per persistent connection. I had set
> number of threads as 16 earlier but looking at code it seemed that by not
> supplying the threads parameter for avro source it uses a flexible thread
> pool, so I removed that setting to let the flexible thread pool be used.
> Still, beyond a certain number of connections I start getting failed
> connections. It would also in general be better if I can work with low
> number of connections to avoid creating too many threads. If I could work
> without sink groups that would be best, I'm curios about the downsides of
> doing that.
>
> Thanks in Advance
>
>
> On Wed, Aug 14, 2013 at 12:59 PM, Pankaj Gupta <pa...@brightroll.com>wrote:
>
>> @Hari
>>
>> You're right, removing the groups seems to have improved performance. I
>> didn't realize that putting sinks in a load balanced group would have this
>> effect on performance. I will be monitor it more and let you know how that
>> goes. Thanks a lot for your help.
>>
>>
>> On Wed, Aug 14, 2013 at 12:34 PM, Pankaj Gupta <pa...@brightroll.com>wrote:
>>
>>> Null sink is actually holding up. I hadn't specified the batchSize so
>>> the default of 100 was being picked up. When I set batchSize to 4000 it
>>> started consuming all events and the channels are not filling up. So it
>>> seems that the problem is a combination of AvroSink with FileChannel.
>>> Memory Channel with Avro Sink works fine and FileChannel with null sink
>>> works fine.
>>>
>>>
>>> On Wed, Aug 14, 2013 at 12:12 PM, Pankaj Gupta <pa...@brightroll.com>wrote:
>>>
>>>> With Null Sink, in the call stack I see a lot of these:
>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>> tid=0x00007feb18001000 nid=0x356b runnable [0x00007feb9e897000]
>>>>     java.lang.Thread.State: RUNNABLE
>>>>         at sun.nio.ch.FileChannelImpl.force0(Native Method)
>>>>         at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:348)
>>>>         at
>>>> org.apache.flume.channel.file.LogFile$Writer.sync(LogFile.java:258)
>>>>         at
>>>> org.apache.flume.channel.file.LogFile$Writer.commit(LogFile.java:225)
>>>>         - locked <0x0000000518c97650> (a
>>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>>>         at org.apache.flume.channel.file.Log.commit(Log.java:758)
>>>>         at org.apache.flume.channel.file.Log.commitTake(Log.java:640)
>>>>         at
>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:557)
>>>>         at
>>>> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
>>>>         at org.apache.flume.sink.NullSink.process(NullSink.java:100)
>>>>         at
>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>         at
>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Aug 14, 2013 at 11:57 AM, Pankaj Gupta <pa...@brightroll.com>wrote:
>>>>
>>>>> I tried increasing the dataDirs to 2 and 4 per disk but doesn't seem
>>>>> to help much. I then replaced the avro sink with null sinks and events are
>>>>> still filling up in the channel. I tried with both 2 and 4 dataDirs per
>>>>> disk and null sink, still don't get throughput higher than 1.5 MBps.
>>>>>
>>>>>
>>>>> On Tue, Aug 13, 2013 at 8:30 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>
>>>>>> Increasing the number of file channels will result in more
>>>>>> checkpoints. Therefore there will be more io than simply increasing the
>>>>>> number of dataDirs.  However, this might be a case where it'd be nice to
>>>>>> relax the file channel data consistency constraints a little to get
>>>>>> increased throughput. That feature does not exist at present.
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 13, 2013 at 10:16 PM, Pankaj Gupta <pankaj@brightroll.com
>>>>>> > wrote:
>>>>>>
>>>>>>> I did try increasing number of FileChannels. At 2 FileChannels per
>>>>>>> disk performance seemed to be 25% better. At 4 FileChannels per disk
>>>>>>> performance dropped to even below 1 FileChannel per disk. I will try
>>>>>>> increasing the dataDirs tomorrow.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 13, 2013 at 8:06 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>>>
>>>>>>>> dataDirs is a comma separated list. Try 3-4 directories and then
>>>>>>>> the same test.
>>>>>>>> On Aug 13, 2013 9:58 PM, "Pankaj Gupta" <pa...@brightroll.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Both disks were at around 15-25%.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Aug 13, 2013 at 7:54 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>>>>>
>>>>>>>>>> Gotcha. When you run tge test what is tye disk utilization
>>>>>>>>>> percentage? Iostat can be used for this.
>>>>>>>>>> On Aug 13, 2013 9:47 PM, "Pankaj Gupta" <pa...@brightroll.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Those are the boxes we want to collect data from. They run flume
>>>>>>>>>>> and send data through their avro sinks to the avro source on this box. We
>>>>>>>>>>> are getting data at a pretty good rate and the problem is in fact that the
>>>>>>>>>>> events don't drain from the FileChannel fast enough and the channel fill
>>>>>>>>>>> percentage keeps getting higher.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 13, 2013 at 7:41 PM, Brock Noland <
>>>>>>>>>>> brock@cloudera.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> What is sending the events to the avro source?
>>>>>>>>>>>> On Aug 13, 2013 9:34 PM, "Pankaj Gupta" <pa...@brightroll.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Here's the config:
>>>>>>>>>>>>> # define channels, one for each disk
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> agent1.channels.ch1.type = FILE
>>>>>>>>>>>>> agent1.channels.ch1.checkpointDir = /flume1/checkpoint
>>>>>>>>>>>>> agent1.channels.ch1.dataDirs = /flume1/data
>>>>>>>>>>>>> agent1.channels.ch1.maxFileSize = 375809638400
>>>>>>>>>>>>> agent1.channels.ch1.capacity = 75000000
>>>>>>>>>>>>> agent1.channels.ch1.transactionCapacity = 4000
>>>>>>>>>>>>>
>>>>>>>>>>>>> agent1.channels.ch2.type = FILE
>>>>>>>>>>>>> agent1.channels.ch2.checkpointDir = /flume2/checkpoint
>>>>>>>>>>>>> agent1.channels.ch2.dataDirs = /flume2/data
>>>>>>>>>>>>> agent1.channels.ch2.maxFileSize = 375809638400
>>>>>>>>>>>>> agent1.channels.ch2.capacity = 75000000
>>>>>>>>>>>>> agent1.channels.ch2.transactionCapacity = 4000
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> # Define an Avro source named avroSource1
>>>>>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> agent1.sources.avroSource1.channels = ch1
>>>>>>>>>>>>> agent1.sources.avroSource1.type = avro
>>>>>>>>>>>>> agent1.sources.avroSource1.bind = 0.0.0.0
>>>>>>>>>>>>> agent1.sources.avroSource1.port = <port>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> agent1.sinks.avroSink1-1-1.type = avro
>>>>>>>>>>>>> agent1.sinks.avroSink1-1-1.channel = ch1
>>>>>>>>>>>>> agent1.sinks.avroSink1-1-1.hostname = <hostname>
>>>>>>>>>>>>> agent1.sinks.avroSink1-1-1.port = <port>
>>>>>>>>>>>>> agent1.sinks.avroSink1-1-1.connect-timeout = 300000
>>>>>>>>>>>>> agent1.sinks.avroSink1-1-1.batch-size = 4000
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> agent1.sinks.avroSink1-2-1.type = avro
>>>>>>>>>>>>> agent1.sinks.avroSink1-2-1.channel = ch1
>>>>>>>>>>>>> agent1.sinks.avroSink1-2-1.hostname = <hostname>
>>>>>>>>>>>>> agent1.sinks.avroSink1-2-1.port = <port>
>>>>>>>>>>>>> agent1.sinks.avroSink1-2-1.connect-timeout = 300000
>>>>>>>>>>>>> agent1.sinks.avroSink1-2-1.batch-size = 4000
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> agent1.sinks.avroSink1-3-1.type = avro
>>>>>>>>>>>>> agent1.sinks.avroSink1-3-1.channel = ch1
>>>>>>>>>>>>> agent1.sinks.avroSink1-3-1.hostname = <hostname>
>>>>>>>>>>>>> agent1.sinks.avroSink1-3-1.port = <port>
>>>>>>>>>>>>> agent1.sinks.avroSink1-3-1.connect-timeout = 300000
>>>>>>>>>>>>> agent1.sinks.avroSink1-3-1.batch-size = 4000
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> agent1.sinks.avroSink1-4-1.type = avro
>>>>>>>>>>>>> agent1.sinks.avroSink1-4-1.channel = ch1
>>>>>>>>>>>>> agent1.sinks.avroSink1-4-1.hostname = <hostname>
>>>>>>>>>>>>> agent1.sinks.avroSink1-4-1.port = <port>
>>>>>>>>>>>>> agent1.sinks.avroSink1-4-1.connect-timeout = 300000
>>>>>>>>>>>>> agent1.sinks.avroSink1-4-1.batch-size = 4000
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>>>>>> which round robin between different hops
>>>>>>>>>>>>> agent1.sinkgroups.group1.sinks = avroSink1-1-1 avroSink1-2-1
>>>>>>>>>>>>> avroSink1-3-1 avroSink1-4-1
>>>>>>>>>>>>> agent1.sinkgroups.group1.processor.type = load_balance
>>>>>>>>>>>>> agent1.sinkgroups.group1.processor.selector = ROUND_ROBIN
>>>>>>>>>>>>> agent1.sinkgroups.group1.processor.backoff = true
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> #End of set
>>>>>>>>>>>>>
>>>>>>>>>>>>> # Define an Avro source named avroSource2
>>>>>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> agent1.sources.avroSource2.channels = ch2
>>>>>>>>>>>>> agent1.sources.avroSource2.type = avro
>>>>>>>>>>>>> agent1.sources.avroSource2.bind = 0.0.0.0
>>>>>>>>>>>>> agent1.sources.avroSource2.port = <port>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> agent1.sinks.avroSink2-1-1.type = avro
>>>>>>>>>>>>> agent1.sinks.avroSink2-1-1.channel = ch2
>>>>>>>>>>>>> agent1.sinks.avroSink2-1-1.hostname = <hostname>
>>>>>>>>>>>>> agent1.sinks.avroSink2-1-1.port = <port>
>>>>>>>>>>>>> agent1.sinks.avroSink2-1-1.connect-timeout = 300000
>>>>>>>>>>>>> agent1.sinks.avroSink2-1-1.batch-size = 4000
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> agent1.sinks.avroSink2-2-1.type = avro
>>>>>>>>>>>>> agent1.sinks.avroSink2-2-1.channel = ch2
>>>>>>>>>>>>> agent1.sinks.avroSink2-2-1.hostname = <hostname>
>>>>>>>>>>>>> agent1.sinks.avroSink2-2-1.port = <port>
>>>>>>>>>>>>> agent1.sinks.avroSink2-2-1.connect-timeout = 300000
>>>>>>>>>>>>> agent1.sinks.avroSink2-2-1.batch-size = 4000
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> agent1.sinks.avroSink2-3-1.type = avro
>>>>>>>>>>>>> agent1.sinks.avroSink2-3-1.channel = ch2
>>>>>>>>>>>>> agent1.sinks.avroSink2-3-1.hostname = <hostname>
>>>>>>>>>>>>> agent1.sinks.avroSink2-3-1.port = <port>
>>>>>>>>>>>>> agent1.sinks.avroSink2-3-1.connect-timeout = 300000
>>>>>>>>>>>>> agent1.sinks.avroSink2-3-1.batch-size = 4000
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> agent1.sinks.avroSink2-4-1.type = avro
>>>>>>>>>>>>> agent1.sinks.avroSink2-4-1.channel = ch2
>>>>>>>>>>>>> agent1.sinks.avroSink2-4-1.hostname = <hostname>
>>>>>>>>>>>>> agent1.sinks.avroSink2-4-1.port = <port>
>>>>>>>>>>>>> agent1.sinks.avroSink2-4-1.connect-timeout = 300000
>>>>>>>>>>>>> agent1.sinks.avroSink2-4-1.batch-size = 4000
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>>>>>> which round robin between different hops
>>>>>>>>>>>>> agent1.sinkgroups.group2.sinks = avroSink2-1-1 avroSink2-2-1
>>>>>>>>>>>>> avroSink2-3-1 avroSink2-4-1
>>>>>>>>>>>>> agent1.sinkgroups.group2.processor.type = load_balance
>>>>>>>>>>>>> agent1.sinkgroups.group2.processor.selector = ROUND_ROBIN
>>>>>>>>>>>>> agent1.sinkgroups.group2.processor.backoff = true
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> #End of set
>>>>>>>>>>>>>
>>>>>>>>>>>>> # Define an Avro source named avroSource3
>>>>>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> agent1.sources.avroSource3.channels = ch1
>>>>>>>>>>>>> agent1.sources.avroSource3.type = avro
>>>>>>>>>>>>> agent1.sources.avroSource3.bind = 0.0.0.0
>>>>>>>>>>>>> agent1.sources.avroSource3.port = <port>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> agent1.sinks.avroSink3-1-1.type = avro
>>>>>>>>>>>>> agent1.sinks.avroSink3-1-1.channel = ch1
>>>>>>>>>>>>> agent1.sinks.avroSink3-1-1.hostname = <hostname>
>>>>>>>>>>>>> agent1.sinks.avroSink3-1-1.port = <port>
>>>>>>>>>>>>> agent1.sinks.avroSink3-1-1.connect-timeout = 300000
>>>>>>>>>>>>> agent1.sinks.avroSink3-1-1.batch-size = 4000
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> agent1.sinks.avroSink3-2-1.type = avro
>>>>>>>>>>>>> agent1.sinks.avroSink3-2-1.channel = ch1
>>>>>>>>>>>>> agent1.sinks.avroSink3-2-1.hostname = <hostname>
>>>>>>>>>>>>> agent1.sinks.avroSink3-2-1.port = <port>
>>>>>>>>>>>>> agent1.sinks.avroSink3-2-1.connect-timeout = 300000
>>>>>>>>>>>>> agent1.sinks.avroSink3-2-1.batch-size = 4000
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> agent1.sinks.avroSink3-3-1.type = avro
>>>>>>>>>>>>> agent1.sinks.avroSink3-3-1.channel = ch1
>>>>>>>>>>>>> agent1.sinks.avroSink3-3-1.hostname = <hostname>
>>>>>>>>>>>>> agent1.sinks.avroSink3-3-1.port = <port>
>>>>>>>>>>>>> agent1.sinks.avroSink3-3-1.connect-timeout = 300000
>>>>>>>>>>>>> agent1.sinks.avroSink3-3-1.batch-size = 4000
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> agent1.sinks.avroSink3-4-1.type = avro
>>>>>>>>>>>>> agent1.sinks.avroSink3-4-1.channel = ch1
>>>>>>>>>>>>> agent1.sinks.avroSink3-4-1.hostname = <hostname>
>>>>>>>>>>>>> agent1.sinks.avroSink3-4-1.port = <port>
>>>>>>>>>>>>> agent1.sinks.avroSink3-4-1.connect-timeout = 300000
>>>>>>>>>>>>> agent1.sinks.avroSink3-4-1.batch-size = 4000
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>>>>>> which round robin between different hops
>>>>>>>>>>>>> agent1.sinkgroups.group3.sinks = avroSink3-1-1 avroSink3-2-1
>>>>>>>>>>>>> avroSink3-3-1 avroSink3-4-1
>>>>>>>>>>>>> agent1.sinkgroups.group3.processor.type = load_balance
>>>>>>>>>>>>> agent1.sinkgroups.group3.processor.selector = ROUND_ROBIN
>>>>>>>>>>>>> agent1.sinkgroups.group3.processor.backoff = true
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> #End of set
>>>>>>>>>>>>>
>>>>>>>>>>>>> # Define an Avro source named avroSource4
>>>>>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> agent1.sources.avroSource4.channels = ch2
>>>>>>>>>>>>> agent1.sources.avroSource4.type = avro
>>>>>>>>>>>>> agent1.sources.avroSource4.bind = 0.0.0.0
>>>>>>>>>>>>> agent1.sources.avroSource4.port = <port>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> agent1.sinks.avroSink4-1-1.type = avro
>>>>>>>>>>>>> agent1.sinks.avroSink4-1-1.channel = ch2
>>>>>>>>>>>>> agent1.sinks.avroSink4-1-1.hostname = <hostname>
>>>>>>>>>>>>> agent1.sinks.avroSink4-1-1.port = <port>
>>>>>>>>>>>>> agent1.sinks.avroSink4-1-1.connect-timeout = 300000
>>>>>>>>>>>>> agent1.sinks.avroSink4-1-1.batch-size = 4000
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> agent1.sinks.avroSink4-2-1.type = avro
>>>>>>>>>>>>> agent1.sinks.avroSink4-2-1.channel = ch2
>>>>>>>>>>>>> agent1.sinks.avroSink4-2-1.hostname = <hostname>
>>>>>>>>>>>>> agent1.sinks.avroSink4-2-1.port = <port>
>>>>>>>>>>>>> agent1.sinks.avroSink4-2-1.connect-timeout = 300000
>>>>>>>>>>>>> agent1.sinks.avroSink4-2-1.batch-size = 4000
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> agent1.sinks.avroSink4-3-1.type = avro
>>>>>>>>>>>>> agent1.sinks.avroSink4-3-1.channel = ch2
>>>>>>>>>>>>> agent1.sinks.avroSink4-3-1.hostname = <hostname>
>>>>>>>>>>>>> agent1.sinks.avroSink4-3-1.port = <port>
>>>>>>>>>>>>> agent1.sinks.avroSink4-3-1.connect-timeout = 300000
>>>>>>>>>>>>> agent1.sinks.avroSink4-3-1.batch-size = 4000
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> agent1.sinks.avroSink4-4-1.type = avro
>>>>>>>>>>>>> agent1.sinks.avroSink4-4-1.channel = ch2
>>>>>>>>>>>>> agent1.sinks.avroSink4-4-1.hostname = <hostname>
>>>>>>>>>>>>> agent1.sinks.avroSink4-4-1.port = <port>
>>>>>>>>>>>>> agent1.sinks.avroSink4-4-1.connect-timeout = 300000
>>>>>>>>>>>>> agent1.sinks.avroSink4-4-1.batch-size = 4000
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>>>>>> which round robin between different hops
>>>>>>>>>>>>> agent1.sinkgroups.group4.sinks = avroSink4-1-1 avroSink4-2-1
>>>>>>>>>>>>> avroSink4-3-1 avroSink4-4-1
>>>>>>>>>>>>> agent1.sinkgroups.group4.processor.type = load_balance
>>>>>>>>>>>>> agent1.sinkgroups.group4.processor.selector = ROUND_ROBIN
>>>>>>>>>>>>> agent1.sinkgroups.group4.processor.backoff = true
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> #End of set
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> # Finally, now that we've defined all of our components, tell
>>>>>>>>>>>>> # agent1 which ones we want to activate.
>>>>>>>>>>>>> agent1.channels = ch1 ch2
>>>>>>>>>>>>> agent1.sources = avroSource1 avroSource2 avroSource3
>>>>>>>>>>>>> avroSource4
>>>>>>>>>>>>> agent1.sinks = avroSink1-1-1 avroSink1-2-1 avroSink1-3-1
>>>>>>>>>>>>> avroSink1-4-1 avroSink2-1-1 avroSink2-2-1 avroSink2-3-1 avroSink2-4-1
>>>>>>>>>>>>> avroSink3-1-1 avroSink3-2-1 avroSink3-3-1 avroSink3-4-1 avroSink4-1-1
>>>>>>>>>>>>> avroSink4-2-1 avroSink4-3-1 avroSink4-4-1
>>>>>>>>>>>>> agent1.sinkgroups = group1 group2 group3 group4
>>>>>>>>>>>>>
>>>>>>>>>>>>> There are two disks on the box /flume1 and /flume2
>>>>>>>>>>>>> 32 cores of CPU, 32 GB RAM, 1GbPS ethernet.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Aug 13, 2013 at 7:22 PM, Brock Noland <
>>>>>>>>>>>>> brock@cloudera.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can you share your conf file?
>>>>>>>>>>>>>> On Aug 13, 2013 9:19 PM, "Hari Shreedharan" <
>>>>>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  Even though the writes are done per batch, they don't go to
>>>>>>>>>>>>>>> disk rightaway - commits are the only ones which actually cause an fsync -
>>>>>>>>>>>>>>> which is when writes actually go to disk.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Hari
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tuesday, August 13, 2013 at 7:06 PM, Pankaj Gupta wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Looking at the code it seems like the lock and the i/o is
>>>>>>>>>>>>>>> done per event and not for a batch. Is that correct? If that is the case
>>>>>>>>>>>>>>> then it seems like there is a lot of overhead per event. The throughput I'm
>>>>>>>>>>>>>>> seeing is 1 - 1.5 MBps per disk which is way below the sequential
>>>>>>>>>>>>>>> read/write capacity of the disk which is easily over 50MBps. Adding more
>>>>>>>>>>>>>>> sinks doesn't help, they just block waiting for the queue to become free.
>>>>>>>>>>>>>>> CPU usage is 20%, there is enough RAM for page cache so that no read is
>>>>>>>>>>>>>>> going to disk. The queue seems to be the bottleneck. What is the throughput
>>>>>>>>>>>>>>> I should expect per disk?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland <
>>>>>>>>>>>>>>> brock@cloudera.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The lock is per file. Adding more directories to the channel
>>>>>>>>>>>>>>> will cause more files to be created. Of course you'll need additional disks
>>>>>>>>>>>>>>> behind those directories to see any performance increase.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <
>>>>>>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  Yes, IO is done inside locks to avoid multiple takes and
>>>>>>>>>>>>>>> puts getting written out at the same time. Even though Java makes sure the
>>>>>>>>>>>>>>> writes are serialized, Flume still needs to keep track of some counters
>>>>>>>>>>>>>>> etc, so the lock is required. Note that the lock you are talking about is
>>>>>>>>>>>>>>>  in the LogFile class, which represents a single file - so even if the
>>>>>>>>>>>>>>> write is inside that lock (which is also inside that class itself) that
>>>>>>>>>>>>>>>  does not cause any contention - because the lock is just preventing two IO
>>>>>>>>>>>>>>> ops to happen at the same time.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Hari
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It seems like some i/o is done inside the lock, which means
>>>>>>>>>>>>>>> that time for taking a lock is proportional to the time for i/o and thus it
>>>>>>>>>>>>>>> becomes a problem. I apologize in advance if I am wrong but the call stack
>>>>>>>>>>>>>>> and behavior I'm seeing seems to suggest that. Specifically, it seems that
>>>>>>>>>>>>>>> we do a write while inside take:
>>>>>>>>>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor"
>>>>>>>>>>>>>>> prio=10 tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>>>         at sun.nio.ch.NativeThread.current(Native Method)
>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>> sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>> sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>>>>>>>>>>>>>>>         - locked <0x00000005190ec998> (a java.lang.Object)
>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>> org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>> org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>>>>>>>>>>>>>>>         - locked <0x0000000519111590> (a
>>>>>>>>>>>>>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>> org.apache.flume.channel.file.Log.take(Log.java:550)
>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>>>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <
>>>>>>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Since the channel is designed to make sure that events are
>>>>>>>>>>>>>>> not duplicated to multiple sinks, and to protect against corruption due to
>>>>>>>>>>>>>>> concurrency issues, we do not need the locking in the channel's flume event
>>>>>>>>>>>>>>> queue. It is unlikely that locking is what is causing performance issues
>>>>>>>>>>>>>>> because the channel is heavily I/O bound. If you take a series of thread
>>>>>>>>>>>>>>> dumps, you will probably see that those threads are moving forward and the
>>>>>>>>>>>>>>> ones reading/writing from/to disk are the ones which are slower. These
>>>>>>>>>>>>>>> locks are unlikely to hit performance much.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Hari
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Spent some more time debugging issues with FileChannel. The
>>>>>>>>>>>>>>> problem seems to lock contention reading from FlumeEventQueue:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I see a lot of threads like this:
>>>>>>>>>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor"
>>>>>>>>>>>>>>> prio=10 tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
>>>>>>>>>>>>>>> [0x00007f821afee000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>>>>>>>>>>>>>>>         - waiting to lock <0x0000000518ee4c90> (a
>>>>>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>>>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have two file channels and 8 Avro Sinks per file channel.
>>>>>>>>>>>>>>> I added more sinks because they weren't draining fast enough. It seems like
>>>>>>>>>>>>>>> they send the batch then wait for an ack before sending again, thus sends
>>>>>>>>>>>>>>> are not pipelined and having more sinks seemed like a good way of getting
>>>>>>>>>>>>>>> some parallelism.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Here's the full stack trace:
>>>>>>>>>>>>>>> 2013-08-13 15:30:32
>>>>>>>>>>>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM
>>>>>>>>>>>>>>> (20.13-b02 mixed mode):
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>>> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: WAITING (parking)
>>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>>> - parking to wait for  <0x000000056d932120> (a
>>>>>>>>>>>>>>> java.util.concurrent.CountDownLatch$Sync)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
>>>>>>>>>>>>>>> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.apache.avro.ipc.Requestor.request(Requestor.java:150)
>>>>>>>>>>>>>>> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
>>>>>>>>>>>>>>> at $Proxy7.appendBatch(Unknown Source)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000
>>>>>>>>>>>>>>> nid=0x41a1 waiting on condition [0x0000000000000000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>>> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>>> - parking to wait for  <0x00000005194c3340> (a
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>>> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>>> - parking to wait for  <0x00000005191a16d8> (a
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>>> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>>> - parking to wait for  <0x0000000519174e08> (a
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>>> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>>> - parking to wait for  <0x0000000519174e98> (a
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>>> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>>> - parking to wait for  <0x00000005195f9670> (a
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>>> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>>> - parking to wait for  <0x0000000519174ce8> (a
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>>> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>>> - parking to wait for  <0x00000005191a1918> (a
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>>> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>>> - parking to wait for  <0x00000005191e7f60> (a
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800
>>>>>>>>>>>>>>> nid=0x4153 runnable [0x00007f8210e4d000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>>  - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>>> - locked <0x000000054a2917a8> (a
>>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>>  - locked <0x000000054a2c5348> (a
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>>> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>>> - parking to wait for  <0x0000000519174f58> (a
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "pool-142-thread-41" prio=10 tid=0x00007f8584469800
>>>>>>>>>>>>>>> nid=0x4146 runnable [0x00007f8210f4e000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>>  - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>>> - locked <0x0000000546e5dd28> (a
>>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>>  - locked <0x0000000546ec1ea0> (a
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "pool-140-thread-41" prio=10 tid=0x00007f8588411000
>>>>>>>>>>>>>>> nid=0x4118 runnable [0x00007f821104f000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>>  - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>>> - locked <0x0000000544f763c0> (a
>>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>>  - locked <0x0000000544ee55f8> (a
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000
>>>>>>>>>>>>>>> nid=0x4111 runnable [0x00007f8211150000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>>  - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>>> - locked <0x000000054473d5a8> (a
>>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>>  - locked <0x000000054474d088> (a
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800
>>>>>>>>>>>>>>> nid=0x4110 runnable [0x00007f8211251000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>>  - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>>> - locked <0x00000005437f4f70> (a
>>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>>  - locked <0x0000000543803478> (a
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "pool-140-thread-40" prio=10 tid=0x00007f858840f800
>>>>>>>>>>>>>>> nid=0x410f runnable [0x00007f8211352000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>>  - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>>> - locked <0x00000005422a9018> (a
>>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>>  - locked <0x00000005422ca310> (a
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000
>>>>>>>>>>>>>>> nid=0x410e runnable [0x00007f8211453000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>>  - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>>> - locked <0x0000000541063d50> (a
>>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>>  - locked <0x00000005410c8240> (a
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000
>>>>>>>>>>>>>>> nid=0x410d runnable [0x00007f8211554000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>>  - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>>> - locked <0x0000000540dc1110> (a
>>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>>  - locked <0x0000000540c94928> (a
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800
>>>>>>>>>>>>>>> nid=0x410a runnable [0x00007f8211655000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>>  - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>>> - locked <0x000000053fbbc3b0> (a
>>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>>  - locked <0x000000053f897d00> (a
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800
>>>>>>>>>>>>>>> nid=0x4109 runnable [0x00007f8211756000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>>  - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>>> - locked <0x000000053f812860> (a
>>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>>  - locked <0x000000053f897cb8> (a
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000
>>>>>>>>>>>>>>> nid=0x40fd runnable [0x00007f8211857000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>>  - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>>> - locked <0x000000053e6e5570> (a
>>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>>  - locked <0x000000053e714e70> (a
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>>> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>>> - parking to wait for  <0x0000000519175138> (a
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>>> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>>> - parking to wait for  <0x0000000519174c28> (a
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>>> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>>> - parking to wait for  <0x00000005191751c8> (a
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800
>>>>>>>>>>>>>>> nid=0x40f0 runnable [0x00007f8211c5b000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>>  - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>>> - locked <0x0000000536579db0> (a
>>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>>  - locked <0x0000000536585830> (a
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>>> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>>> - parking to wait for  <0x00000005191750a8> (a
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000
>>>>>>>>>>>>>>> nid=0x40e1 runnable [0x00007f8211e5d000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>>  - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>>> - locked <0x00000005313943a8> (a
>>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>>  - locked <0x00000005314e0fe0> (a
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>>> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>>> - parking to wait for  <0x000000051936f460> (a
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>>> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>>> - parking to wait for  <0x000000051936dfa8> (a
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800
>>>>>>>>>>>>>>> nid=0x40de runnable [0x00007f8212160000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>>  - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>>> - locked <0x000000052dad4468> (a
>>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>>  - locked <0x000000052da31528> (a
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800
>>>>>>>>>>>>>>> nid=0x40dd runnable [0x00007f8212261000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>>  - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>>> - locked <0x000000052c73c5f0> (a
>>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>>  - locked <0x000000052c652608> (a
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000
>>>>>>>>>>>>>>> nid=0x40dc runnable [0x00007f8212362000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>>  - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>>> - locked <0x000000052b8da998> (a
>>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>>  - locked <0x000000052b8fc590> (a
>>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000
>>>>>>>>>>>>>>> nid=0x40db waiting for monitor entry [0x00007f8212462000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>>>>>>>>>>>>>>> - waiting to lock <0x0000000518ee4c90> (a
>>>>>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>>>>>>>>>>>>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown
>>>>>>>>>>>>>>> Source)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>>>>>>>>>>>>>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>>> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>>> - parking to wait for  <0x000000051936df18> (a
>>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ...
>
> [Message clipped]




-- 


*P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com

Pankaj Gupta | Software Engineer

*BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com


United States | Canada | United Kingdom | Germany


We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
!

Re: Lock contention in FileChannel

[Pankaj Gupta <pa...@brightroll.com>]

I am getting very good performance by removing the groups altogether and
keeping a high number of total sinks (64). If I organize sinks into groups
of 4 I loose parallelism and don't get as good performance as with 64
sinks. The reason I want to organize sinks into groups is for failover. Do
I get any failover if I don't use any groups? E.g. if I have 64 sinks and
one of them fails to send events, would some other sink be able to send
those events or would the failed sink keep trying forever.

The reason I am not able to increase the number of sinks beyond 64 is that
I start getting failed connection exceptions. It seem like the avro source
on the destination is not allowing connections beyond a certain number. It
seems that avro sink creates a thread per persistent connection. I had set
number of threads as 16 earlier but looking at code it seemed that by not
supplying the threads parameter for avro source it uses a flexible thread
pool, so I removed that setting to let the flexible thread pool be used.
Still, beyond a certain number of connections I start getting failed
connections. It would also in general be better if I can work with low
number of connections to avoid creating too many threads. If I could work
without sink groups that would be best, I'm curios about the downsides of
doing that.

Thanks in Advance


On Wed, Aug 14, 2013 at 12:59 PM, Pankaj Gupta <pa...@brightroll.com>wrote:

> @Hari
>
> You're right, removing the groups seems to have improved performance. I
> didn't realize that putting sinks in a load balanced group would have this
> effect on performance. I will be monitor it more and let you know how that
> goes. Thanks a lot for your help.
>
>
> On Wed, Aug 14, 2013 at 12:34 PM, Pankaj Gupta <pa...@brightroll.com>wrote:
>
>> Null sink is actually holding up. I hadn't specified the batchSize so the
>> default of 100 was being picked up. When I set batchSize to 4000 it started
>> consuming all events and the channels are not filling up. So it seems that
>> the problem is a combination of AvroSink with FileChannel. Memory Channel
>> with Avro Sink works fine and FileChannel with null sink works fine.
>>
>>
>> On Wed, Aug 14, 2013 at 12:12 PM, Pankaj Gupta <pa...@brightroll.com>wrote:
>>
>>> With Null Sink, in the call stack I see a lot of these:
>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>> tid=0x00007feb18001000 nid=0x356b runnable [0x00007feb9e897000]
>>>     java.lang.Thread.State: RUNNABLE
>>>         at sun.nio.ch.FileChannelImpl.force0(Native Method)
>>>         at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:348)
>>>         at
>>> org.apache.flume.channel.file.LogFile$Writer.sync(LogFile.java:258)
>>>         at
>>> org.apache.flume.channel.file.LogFile$Writer.commit(LogFile.java:225)
>>>         - locked <0x0000000518c97650> (a
>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>>         at org.apache.flume.channel.file.Log.commit(Log.java:758)
>>>         at org.apache.flume.channel.file.Log.commitTake(Log.java:640)
>>>         at
>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:557)
>>>         at
>>> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
>>>         at org.apache.flume.sink.NullSink.process(NullSink.java:100)
>>>         at
>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>         at
>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>         at java.lang.Thread.run(Thread.java:662)
>>>
>>>
>>>
>>>
>>> On Wed, Aug 14, 2013 at 11:57 AM, Pankaj Gupta <pa...@brightroll.com>wrote:
>>>
>>>> I tried increasing the dataDirs to 2 and 4 per disk but doesn't seem to
>>>> help much. I then replaced the avro sink with null sinks and events are
>>>> still filling up in the channel. I tried with both 2 and 4 dataDirs per
>>>> disk and null sink, still don't get throughput higher than 1.5 MBps.
>>>>
>>>>
>>>> On Tue, Aug 13, 2013 at 8:30 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>
>>>>> Increasing the number of file channels will result in more
>>>>> checkpoints. Therefore there will be more io than simply increasing the
>>>>> number of dataDirs.  However, this might be a case where it'd be nice to
>>>>> relax the file channel data consistency constraints a little to get
>>>>> increased throughput. That feature does not exist at present.
>>>>>
>>>>>
>>>>> On Tue, Aug 13, 2013 at 10:16 PM, Pankaj Gupta <pa...@brightroll.com>wrote:
>>>>>
>>>>>> I did try increasing number of FileChannels. At 2 FileChannels per
>>>>>> disk performance seemed to be 25% better. At 4 FileChannels per disk
>>>>>> performance dropped to even below 1 FileChannel per disk. I will try
>>>>>> increasing the dataDirs tomorrow.
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 13, 2013 at 8:06 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>>
>>>>>>> dataDirs is a comma separated list. Try 3-4 directories and then the
>>>>>>> same test.
>>>>>>> On Aug 13, 2013 9:58 PM, "Pankaj Gupta" <pa...@brightroll.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Both disks were at around 15-25%.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Aug 13, 2013 at 7:54 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>>>>
>>>>>>>>> Gotcha. When you run tge test what is tye disk utilization
>>>>>>>>> percentage? Iostat can be used for this.
>>>>>>>>> On Aug 13, 2013 9:47 PM, "Pankaj Gupta" <pa...@brightroll.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Those are the boxes we want to collect data from. They run flume
>>>>>>>>>> and send data through their avro sinks to the avro source on this box. We
>>>>>>>>>> are getting data at a pretty good rate and the problem is in fact that the
>>>>>>>>>> events don't drain from the FileChannel fast enough and the channel fill
>>>>>>>>>> percentage keeps getting higher.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Aug 13, 2013 at 7:41 PM, Brock Noland <brock@cloudera.com
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>> What is sending the events to the avro source?
>>>>>>>>>>> On Aug 13, 2013 9:34 PM, "Pankaj Gupta" <pa...@brightroll.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Here's the config:
>>>>>>>>>>>> # define channels, one for each disk
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> agent1.channels.ch1.type = FILE
>>>>>>>>>>>> agent1.channels.ch1.checkpointDir = /flume1/checkpoint
>>>>>>>>>>>> agent1.channels.ch1.dataDirs = /flume1/data
>>>>>>>>>>>> agent1.channels.ch1.maxFileSize = 375809638400
>>>>>>>>>>>> agent1.channels.ch1.capacity = 75000000
>>>>>>>>>>>> agent1.channels.ch1.transactionCapacity = 4000
>>>>>>>>>>>>
>>>>>>>>>>>> agent1.channels.ch2.type = FILE
>>>>>>>>>>>> agent1.channels.ch2.checkpointDir = /flume2/checkpoint
>>>>>>>>>>>> agent1.channels.ch2.dataDirs = /flume2/data
>>>>>>>>>>>> agent1.channels.ch2.maxFileSize = 375809638400
>>>>>>>>>>>> agent1.channels.ch2.capacity = 75000000
>>>>>>>>>>>> agent1.channels.ch2.transactionCapacity = 4000
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> # Define an Avro source named avroSource1
>>>>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> agent1.sources.avroSource1.channels = ch1
>>>>>>>>>>>> agent1.sources.avroSource1.type = avro
>>>>>>>>>>>> agent1.sources.avroSource1.bind = 0.0.0.0
>>>>>>>>>>>> agent1.sources.avroSource1.port = <port>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> agent1.sinks.avroSink1-1-1.type = avro
>>>>>>>>>>>> agent1.sinks.avroSink1-1-1.channel = ch1
>>>>>>>>>>>> agent1.sinks.avroSink1-1-1.hostname = <hostname>
>>>>>>>>>>>> agent1.sinks.avroSink1-1-1.port = <port>
>>>>>>>>>>>> agent1.sinks.avroSink1-1-1.connect-timeout = 300000
>>>>>>>>>>>> agent1.sinks.avroSink1-1-1.batch-size = 4000
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> agent1.sinks.avroSink1-2-1.type = avro
>>>>>>>>>>>> agent1.sinks.avroSink1-2-1.channel = ch1
>>>>>>>>>>>> agent1.sinks.avroSink1-2-1.hostname = <hostname>
>>>>>>>>>>>> agent1.sinks.avroSink1-2-1.port = <port>
>>>>>>>>>>>> agent1.sinks.avroSink1-2-1.connect-timeout = 300000
>>>>>>>>>>>> agent1.sinks.avroSink1-2-1.batch-size = 4000
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> agent1.sinks.avroSink1-3-1.type = avro
>>>>>>>>>>>> agent1.sinks.avroSink1-3-1.channel = ch1
>>>>>>>>>>>> agent1.sinks.avroSink1-3-1.hostname = <hostname>
>>>>>>>>>>>> agent1.sinks.avroSink1-3-1.port = <port>
>>>>>>>>>>>> agent1.sinks.avroSink1-3-1.connect-timeout = 300000
>>>>>>>>>>>> agent1.sinks.avroSink1-3-1.batch-size = 4000
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> agent1.sinks.avroSink1-4-1.type = avro
>>>>>>>>>>>> agent1.sinks.avroSink1-4-1.channel = ch1
>>>>>>>>>>>> agent1.sinks.avroSink1-4-1.hostname = <hostname>
>>>>>>>>>>>> agent1.sinks.avroSink1-4-1.port = <port>
>>>>>>>>>>>> agent1.sinks.avroSink1-4-1.connect-timeout = 300000
>>>>>>>>>>>> agent1.sinks.avroSink1-4-1.batch-size = 4000
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>>>>> which round robin between different hops
>>>>>>>>>>>> agent1.sinkgroups.group1.sinks = avroSink1-1-1 avroSink1-2-1
>>>>>>>>>>>> avroSink1-3-1 avroSink1-4-1
>>>>>>>>>>>> agent1.sinkgroups.group1.processor.type = load_balance
>>>>>>>>>>>> agent1.sinkgroups.group1.processor.selector = ROUND_ROBIN
>>>>>>>>>>>> agent1.sinkgroups.group1.processor.backoff = true
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> #End of set
>>>>>>>>>>>>
>>>>>>>>>>>> # Define an Avro source named avroSource2
>>>>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> agent1.sources.avroSource2.channels = ch2
>>>>>>>>>>>> agent1.sources.avroSource2.type = avro
>>>>>>>>>>>> agent1.sources.avroSource2.bind = 0.0.0.0
>>>>>>>>>>>> agent1.sources.avroSource2.port = <port>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> agent1.sinks.avroSink2-1-1.type = avro
>>>>>>>>>>>> agent1.sinks.avroSink2-1-1.channel = ch2
>>>>>>>>>>>> agent1.sinks.avroSink2-1-1.hostname = <hostname>
>>>>>>>>>>>> agent1.sinks.avroSink2-1-1.port = <port>
>>>>>>>>>>>> agent1.sinks.avroSink2-1-1.connect-timeout = 300000
>>>>>>>>>>>> agent1.sinks.avroSink2-1-1.batch-size = 4000
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> agent1.sinks.avroSink2-2-1.type = avro
>>>>>>>>>>>> agent1.sinks.avroSink2-2-1.channel = ch2
>>>>>>>>>>>> agent1.sinks.avroSink2-2-1.hostname = <hostname>
>>>>>>>>>>>> agent1.sinks.avroSink2-2-1.port = <port>
>>>>>>>>>>>> agent1.sinks.avroSink2-2-1.connect-timeout = 300000
>>>>>>>>>>>> agent1.sinks.avroSink2-2-1.batch-size = 4000
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> agent1.sinks.avroSink2-3-1.type = avro
>>>>>>>>>>>> agent1.sinks.avroSink2-3-1.channel = ch2
>>>>>>>>>>>> agent1.sinks.avroSink2-3-1.hostname = <hostname>
>>>>>>>>>>>> agent1.sinks.avroSink2-3-1.port = <port>
>>>>>>>>>>>> agent1.sinks.avroSink2-3-1.connect-timeout = 300000
>>>>>>>>>>>> agent1.sinks.avroSink2-3-1.batch-size = 4000
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> agent1.sinks.avroSink2-4-1.type = avro
>>>>>>>>>>>> agent1.sinks.avroSink2-4-1.channel = ch2
>>>>>>>>>>>> agent1.sinks.avroSink2-4-1.hostname = <hostname>
>>>>>>>>>>>> agent1.sinks.avroSink2-4-1.port = <port>
>>>>>>>>>>>> agent1.sinks.avroSink2-4-1.connect-timeout = 300000
>>>>>>>>>>>> agent1.sinks.avroSink2-4-1.batch-size = 4000
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>>>>> which round robin between different hops
>>>>>>>>>>>> agent1.sinkgroups.group2.sinks = avroSink2-1-1 avroSink2-2-1
>>>>>>>>>>>> avroSink2-3-1 avroSink2-4-1
>>>>>>>>>>>> agent1.sinkgroups.group2.processor.type = load_balance
>>>>>>>>>>>> agent1.sinkgroups.group2.processor.selector = ROUND_ROBIN
>>>>>>>>>>>> agent1.sinkgroups.group2.processor.backoff = true
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> #End of set
>>>>>>>>>>>>
>>>>>>>>>>>> # Define an Avro source named avroSource3
>>>>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> agent1.sources.avroSource3.channels = ch1
>>>>>>>>>>>> agent1.sources.avroSource3.type = avro
>>>>>>>>>>>> agent1.sources.avroSource3.bind = 0.0.0.0
>>>>>>>>>>>> agent1.sources.avroSource3.port = <port>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> agent1.sinks.avroSink3-1-1.type = avro
>>>>>>>>>>>> agent1.sinks.avroSink3-1-1.channel = ch1
>>>>>>>>>>>> agent1.sinks.avroSink3-1-1.hostname = <hostname>
>>>>>>>>>>>> agent1.sinks.avroSink3-1-1.port = <port>
>>>>>>>>>>>> agent1.sinks.avroSink3-1-1.connect-timeout = 300000
>>>>>>>>>>>> agent1.sinks.avroSink3-1-1.batch-size = 4000
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> agent1.sinks.avroSink3-2-1.type = avro
>>>>>>>>>>>> agent1.sinks.avroSink3-2-1.channel = ch1
>>>>>>>>>>>> agent1.sinks.avroSink3-2-1.hostname = <hostname>
>>>>>>>>>>>> agent1.sinks.avroSink3-2-1.port = <port>
>>>>>>>>>>>> agent1.sinks.avroSink3-2-1.connect-timeout = 300000
>>>>>>>>>>>> agent1.sinks.avroSink3-2-1.batch-size = 4000
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> agent1.sinks.avroSink3-3-1.type = avro
>>>>>>>>>>>> agent1.sinks.avroSink3-3-1.channel = ch1
>>>>>>>>>>>> agent1.sinks.avroSink3-3-1.hostname = <hostname>
>>>>>>>>>>>> agent1.sinks.avroSink3-3-1.port = <port>
>>>>>>>>>>>> agent1.sinks.avroSink3-3-1.connect-timeout = 300000
>>>>>>>>>>>> agent1.sinks.avroSink3-3-1.batch-size = 4000
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> agent1.sinks.avroSink3-4-1.type = avro
>>>>>>>>>>>> agent1.sinks.avroSink3-4-1.channel = ch1
>>>>>>>>>>>> agent1.sinks.avroSink3-4-1.hostname = <hostname>
>>>>>>>>>>>> agent1.sinks.avroSink3-4-1.port = <port>
>>>>>>>>>>>> agent1.sinks.avroSink3-4-1.connect-timeout = 300000
>>>>>>>>>>>> agent1.sinks.avroSink3-4-1.batch-size = 4000
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>>>>> which round robin between different hops
>>>>>>>>>>>> agent1.sinkgroups.group3.sinks = avroSink3-1-1 avroSink3-2-1
>>>>>>>>>>>> avroSink3-3-1 avroSink3-4-1
>>>>>>>>>>>> agent1.sinkgroups.group3.processor.type = load_balance
>>>>>>>>>>>> agent1.sinkgroups.group3.processor.selector = ROUND_ROBIN
>>>>>>>>>>>> agent1.sinkgroups.group3.processor.backoff = true
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> #End of set
>>>>>>>>>>>>
>>>>>>>>>>>> # Define an Avro source named avroSource4
>>>>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> agent1.sources.avroSource4.channels = ch2
>>>>>>>>>>>> agent1.sources.avroSource4.type = avro
>>>>>>>>>>>> agent1.sources.avroSource4.bind = 0.0.0.0
>>>>>>>>>>>> agent1.sources.avroSource4.port = <port>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> agent1.sinks.avroSink4-1-1.type = avro
>>>>>>>>>>>> agent1.sinks.avroSink4-1-1.channel = ch2
>>>>>>>>>>>> agent1.sinks.avroSink4-1-1.hostname = <hostname>
>>>>>>>>>>>> agent1.sinks.avroSink4-1-1.port = <port>
>>>>>>>>>>>> agent1.sinks.avroSink4-1-1.connect-timeout = 300000
>>>>>>>>>>>> agent1.sinks.avroSink4-1-1.batch-size = 4000
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> agent1.sinks.avroSink4-2-1.type = avro
>>>>>>>>>>>> agent1.sinks.avroSink4-2-1.channel = ch2
>>>>>>>>>>>> agent1.sinks.avroSink4-2-1.hostname = <hostname>
>>>>>>>>>>>> agent1.sinks.avroSink4-2-1.port = <port>
>>>>>>>>>>>> agent1.sinks.avroSink4-2-1.connect-timeout = 300000
>>>>>>>>>>>> agent1.sinks.avroSink4-2-1.batch-size = 4000
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> agent1.sinks.avroSink4-3-1.type = avro
>>>>>>>>>>>> agent1.sinks.avroSink4-3-1.channel = ch2
>>>>>>>>>>>> agent1.sinks.avroSink4-3-1.hostname = <hostname>
>>>>>>>>>>>> agent1.sinks.avroSink4-3-1.port = <port>
>>>>>>>>>>>> agent1.sinks.avroSink4-3-1.connect-timeout = 300000
>>>>>>>>>>>> agent1.sinks.avroSink4-3-1.batch-size = 4000
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> agent1.sinks.avroSink4-4-1.type = avro
>>>>>>>>>>>> agent1.sinks.avroSink4-4-1.channel = ch2
>>>>>>>>>>>> agent1.sinks.avroSink4-4-1.hostname = <hostname>
>>>>>>>>>>>> agent1.sinks.avroSink4-4-1.port = <port>
>>>>>>>>>>>> agent1.sinks.avroSink4-4-1.connect-timeout = 300000
>>>>>>>>>>>> agent1.sinks.avroSink4-4-1.batch-size = 4000
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>>>>> which round robin between different hops
>>>>>>>>>>>> agent1.sinkgroups.group4.sinks = avroSink4-1-1 avroSink4-2-1
>>>>>>>>>>>> avroSink4-3-1 avroSink4-4-1
>>>>>>>>>>>> agent1.sinkgroups.group4.processor.type = load_balance
>>>>>>>>>>>> agent1.sinkgroups.group4.processor.selector = ROUND_ROBIN
>>>>>>>>>>>> agent1.sinkgroups.group4.processor.backoff = true
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> #End of set
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> # Finally, now that we've defined all of our components, tell
>>>>>>>>>>>> # agent1 which ones we want to activate.
>>>>>>>>>>>> agent1.channels = ch1 ch2
>>>>>>>>>>>> agent1.sources = avroSource1 avroSource2 avroSource3 avroSource4
>>>>>>>>>>>> agent1.sinks = avroSink1-1-1 avroSink1-2-1 avroSink1-3-1
>>>>>>>>>>>> avroSink1-4-1 avroSink2-1-1 avroSink2-2-1 avroSink2-3-1 avroSink2-4-1
>>>>>>>>>>>> avroSink3-1-1 avroSink3-2-1 avroSink3-3-1 avroSink3-4-1 avroSink4-1-1
>>>>>>>>>>>> avroSink4-2-1 avroSink4-3-1 avroSink4-4-1
>>>>>>>>>>>> agent1.sinkgroups = group1 group2 group3 group4
>>>>>>>>>>>>
>>>>>>>>>>>> There are two disks on the box /flume1 and /flume2
>>>>>>>>>>>> 32 cores of CPU, 32 GB RAM, 1GbPS ethernet.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Aug 13, 2013 at 7:22 PM, Brock Noland <
>>>>>>>>>>>> brock@cloudera.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Can you share your conf file?
>>>>>>>>>>>>> On Aug 13, 2013 9:19 PM, "Hari Shreedharan" <
>>>>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>  Even though the writes are done per batch, they don't go to
>>>>>>>>>>>>>> disk rightaway - commits are the only ones which actually cause an fsync -
>>>>>>>>>>>>>> which is when writes actually go to disk.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Hari
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tuesday, August 13, 2013 at 7:06 PM, Pankaj Gupta wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Looking at the code it seems like the lock and the i/o is
>>>>>>>>>>>>>> done per event and not for a batch. Is that correct? If that is the case
>>>>>>>>>>>>>> then it seems like there is a lot of overhead per event. The throughput I'm
>>>>>>>>>>>>>> seeing is 1 - 1.5 MBps per disk which is way below the sequential
>>>>>>>>>>>>>> read/write capacity of the disk which is easily over 50MBps. Adding more
>>>>>>>>>>>>>> sinks doesn't help, they just block waiting for the queue to become free.
>>>>>>>>>>>>>> CPU usage is 20%, there is enough RAM for page cache so that no read is
>>>>>>>>>>>>>> going to disk. The queue seems to be the bottleneck. What is the throughput
>>>>>>>>>>>>>> I should expect per disk?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland <
>>>>>>>>>>>>>> brock@cloudera.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The lock is per file. Adding more directories to the channel
>>>>>>>>>>>>>> will cause more files to be created. Of course you'll need additional disks
>>>>>>>>>>>>>> behind those directories to see any performance increase.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <
>>>>>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  Yes, IO is done inside locks to avoid multiple takes and
>>>>>>>>>>>>>> puts getting written out at the same time. Even though Java makes sure the
>>>>>>>>>>>>>> writes are serialized, Flume still needs to keep track of some counters
>>>>>>>>>>>>>> etc, so the lock is required. Note that the lock you are talking about is
>>>>>>>>>>>>>>  in the LogFile class, which represents a single file - so even if the
>>>>>>>>>>>>>> write is inside that lock (which is also inside that class itself) that
>>>>>>>>>>>>>>  does not cause any contention - because the lock is just preventing two IO
>>>>>>>>>>>>>> ops to happen at the same time.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Hari
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It seems like some i/o is done inside the lock, which means
>>>>>>>>>>>>>> that time for taking a lock is proportional to the time for i/o and thus it
>>>>>>>>>>>>>> becomes a problem. I apologize in advance if I am wrong but the call stack
>>>>>>>>>>>>>> and behavior I'm seeing seems to suggest that. Specifically, it seems that
>>>>>>>>>>>>>> we do a write while inside take:
>>>>>>>>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>>>>>>>>>> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>>         at sun.nio.ch.NativeThread.current(Native Method)
>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>> sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>> sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>>>>>>>>>>>>>>         - locked <0x00000005190ec998> (a java.lang.Object)
>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>> org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>> org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>>>>>>>>>>>>>>         - locked <0x0000000519111590> (a
>>>>>>>>>>>>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>> org.apache.flume.channel.file.Log.take(Log.java:550)
>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <
>>>>>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Since the channel is designed to make sure that events are
>>>>>>>>>>>>>> not duplicated to multiple sinks, and to protect against corruption due to
>>>>>>>>>>>>>> concurrency issues, we do not need the locking in the channel's flume event
>>>>>>>>>>>>>> queue. It is unlikely that locking is what is causing performance issues
>>>>>>>>>>>>>> because the channel is heavily I/O bound. If you take a series of thread
>>>>>>>>>>>>>> dumps, you will probably see that those threads are moving forward and the
>>>>>>>>>>>>>> ones reading/writing from/to disk are the ones which are slower. These
>>>>>>>>>>>>>> locks are unlikely to hit performance much.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Hari
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Spent some more time debugging issues with FileChannel. The
>>>>>>>>>>>>>> problem seems to lock contention reading from FlumeEventQueue:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I see a lot of threads like this:
>>>>>>>>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>>>>>>>>>> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
>>>>>>>>>>>>>> [0x00007f821afee000]
>>>>>>>>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>>>>>>>>>>>>>>         - waiting to lock <0x0000000518ee4c90> (a
>>>>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have two file channels and 8 Avro Sinks per file channel. I
>>>>>>>>>>>>>> added more sinks because they weren't draining fast enough. It seems like
>>>>>>>>>>>>>> they send the batch then wait for an ack before sending again, thus sends
>>>>>>>>>>>>>> are not pipelined and having more sinks seemed like a good way of getting
>>>>>>>>>>>>>> some parallelism.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here's the full stack trace:
>>>>>>>>>>>>>> 2013-08-13 15:30:32
>>>>>>>>>>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02
>>>>>>>>>>>>>> mixed mode):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
>>>>>>>>>>>>>>    java.lang.Thread.State: WAITING (parking)
>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>> - parking to wait for  <0x000000056d932120> (a
>>>>>>>>>>>>>> java.util.concurrent.CountDownLatch$Sync)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
>>>>>>>>>>>>>> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
>>>>>>>>>>>>>>  at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
>>>>>>>>>>>>>> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
>>>>>>>>>>>>>> at $Proxy7.appendBatch(Unknown Source)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000
>>>>>>>>>>>>>> nid=0x41a1 waiting on condition [0x0000000000000000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>> - parking to wait for  <0x00000005194c3340> (a
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>> - parking to wait for  <0x00000005191a16d8> (a
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>> - parking to wait for  <0x0000000519174e08> (a
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>> - parking to wait for  <0x0000000519174e98> (a
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>> - parking to wait for  <0x00000005195f9670> (a
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>> - parking to wait for  <0x0000000519174ce8> (a
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>> - parking to wait for  <0x00000005191a1918> (a
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>> - parking to wait for  <0x00000005191e7f60> (a
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800
>>>>>>>>>>>>>> nid=0x4153 runnable [0x00007f8210e4d000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>  - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>> - locked <0x000000054a2917a8> (a
>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>  - locked <0x000000054a2c5348> (a
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>> - parking to wait for  <0x0000000519174f58> (a
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "pool-142-thread-41" prio=10 tid=0x00007f8584469800
>>>>>>>>>>>>>> nid=0x4146 runnable [0x00007f8210f4e000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>  - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>> - locked <0x0000000546e5dd28> (a
>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>  - locked <0x0000000546ec1ea0> (a
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "pool-140-thread-41" prio=10 tid=0x00007f8588411000
>>>>>>>>>>>>>> nid=0x4118 runnable [0x00007f821104f000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>  - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>> - locked <0x0000000544f763c0> (a
>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>  - locked <0x0000000544ee55f8> (a
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000
>>>>>>>>>>>>>> nid=0x4111 runnable [0x00007f8211150000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>  - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>> - locked <0x000000054473d5a8> (a
>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>  - locked <0x000000054474d088> (a
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800
>>>>>>>>>>>>>> nid=0x4110 runnable [0x00007f8211251000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>  - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>> - locked <0x00000005437f4f70> (a
>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>  - locked <0x0000000543803478> (a
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "pool-140-thread-40" prio=10 tid=0x00007f858840f800
>>>>>>>>>>>>>> nid=0x410f runnable [0x00007f8211352000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>  - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>> - locked <0x00000005422a9018> (a
>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>  - locked <0x00000005422ca310> (a
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000
>>>>>>>>>>>>>> nid=0x410e runnable [0x00007f8211453000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>  - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>> - locked <0x0000000541063d50> (a
>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>  - locked <0x00000005410c8240> (a
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000
>>>>>>>>>>>>>> nid=0x410d runnable [0x00007f8211554000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>  - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>> - locked <0x0000000540dc1110> (a
>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>  - locked <0x0000000540c94928> (a
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800
>>>>>>>>>>>>>> nid=0x410a runnable [0x00007f8211655000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>  - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>> - locked <0x000000053fbbc3b0> (a
>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>  - locked <0x000000053f897d00> (a
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800
>>>>>>>>>>>>>> nid=0x4109 runnable [0x00007f8211756000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>  - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>> - locked <0x000000053f812860> (a
>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>  - locked <0x000000053f897cb8> (a
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000
>>>>>>>>>>>>>> nid=0x40fd runnable [0x00007f8211857000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>  - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>> - locked <0x000000053e6e5570> (a
>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>  - locked <0x000000053e714e70> (a
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>> - parking to wait for  <0x0000000519175138> (a
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>> - parking to wait for  <0x0000000519174c28> (a
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>> - parking to wait for  <0x00000005191751c8> (a
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800
>>>>>>>>>>>>>> nid=0x40f0 runnable [0x00007f8211c5b000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>  - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>> - locked <0x0000000536579db0> (a
>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>  - locked <0x0000000536585830> (a
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>> - parking to wait for  <0x00000005191750a8> (a
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000
>>>>>>>>>>>>>> nid=0x40e1 runnable [0x00007f8211e5d000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>  - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>> - locked <0x00000005313943a8> (a
>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>  - locked <0x00000005314e0fe0> (a
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>> - parking to wait for  <0x000000051936f460> (a
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>> - parking to wait for  <0x000000051936dfa8> (a
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800
>>>>>>>>>>>>>> nid=0x40de runnable [0x00007f8212160000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>  - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>> - locked <0x000000052dad4468> (a
>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>  - locked <0x000000052da31528> (a
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800
>>>>>>>>>>>>>> nid=0x40dd runnable [0x00007f8212261000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>  - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>> - locked <0x000000052c73c5f0> (a
>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>  - locked <0x000000052c652608> (a
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000
>>>>>>>>>>>>>> nid=0x40dc runnable [0x00007f8212362000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>  - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>> - locked <0x000000052b8da998> (a
>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>  - locked <0x000000052b8fc590> (a
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000
>>>>>>>>>>>>>> nid=0x40db waiting for monitor entry [0x00007f8212462000]
>>>>>>>>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>>>>>>>>>>>>>> - waiting to lock <0x0000000518ee4c90> (a
>>>>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>>>>>>>>>>>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>>>>>>>>>>>>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>> - parking to wait for  <0x000000051936df18> (a
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>>> tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000]
>>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>>> - parking to wait for  <0x0000000518fe7ae8> (a
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000
>>>>>>>>>>>>>> nid=0x40d8 runnable [0x00007f8212766000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>  - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>> - locked <0x00000005263d6ba8> (a
>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>  - locked <0x00000005264b1a48> (a
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800
>>>>>>>>>>>>>> nid=0x40d7 runnable [0x00007f8212867000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>  - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>> - locked <0x0000000525d1d0e0> (a
>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>  - locked <0x0000000525d2cdf8> (a
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000
>>>>>>>>>>>>>> nid=0x40d6 runnable [0x00007f8212968000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>  - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>> - locked <0x0000000525a689d8> (a
>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>  - locked <0x0000000525a24570> (a
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000
>>>>>>>>>>>>>> nid=0x40d5 runnable [0x00007f8212a69000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>  - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>> - locked <0x0000000524e1e7d8> (a
>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>  - locked <0x0000000524cfd670> (a
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800
>>>>>>>>>>>>>> nid=0x40d4 runnable [0x00007f8212b6a000]
>>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>>  - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>>> - locked <0x0000000524d437a8> (a
>>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>>  - locked <0x0000000524ab3330> (a
>>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ...
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>>
>>>>>> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 |
>>>>>> pankaj@brightroll.com
>>>>>>
>>>>>> Pankaj Gupta | Software Engineer
>>>>>>
>>>>>> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>>>>>>
>>>>>>
>>>>>> United States | Canada | United Kingdom | Germany
>>>>>>
>>>>>>
>>>>>> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
>>>>>> !
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>>
>>>> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 |
>>>> pankaj@brightroll.com
>>>>
>>>> Pankaj Gupta | Software Engineer
>>>>
>>>> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>>>>
>>>>
>>>> United States | Canada | United Kingdom | Germany
>>>>
>>>>
>>>> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
>>>> !
>>>>
>>>
>>>
>>>
>>> --
>>>
>>>
>>> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 |
>>> pankaj@brightroll.com
>>>
>>> Pankaj Gupta | Software Engineer
>>>
>>> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>>>
>>>
>>> United States | Canada | United Kingdom | Germany
>>>
>>>
>>> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
>>> !
>>>
>>
>>
>>
>> --
>>
>>
>> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 |
>> pankaj@brightroll.com
>>
>> Pankaj Gupta | Software Engineer
>>
>> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>>
>>
>> United States | Canada | United Kingdom | Germany
>>
>>
>> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
>> !
>>
>
>
>
> --
>
>
> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com
>
> Pankaj Gupta | Software Engineer
>
> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>
>
> United States | Canada | United Kingdom | Germany
>
>
> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
> !
>



-- 


*P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com

Pankaj Gupta | Software Engineer

*BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com


United States | Canada | United Kingdom | Germany


We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
!

Re: Lock contention in FileChannel

[Pankaj Gupta <pa...@brightroll.com>]

@Hari

You're right, removing the groups seems to have improved performance. I
didn't realize that putting sinks in a load balanced group would have this
effect on performance. I will be monitor it more and let you know how that
goes. Thanks a lot for your help.


On Wed, Aug 14, 2013 at 12:34 PM, Pankaj Gupta <pa...@brightroll.com>wrote:

> Null sink is actually holding up. I hadn't specified the batchSize so the
> default of 100 was being picked up. When I set batchSize to 4000 it started
> consuming all events and the channels are not filling up. So it seems that
> the problem is a combination of AvroSink with FileChannel. Memory Channel
> with Avro Sink works fine and FileChannel with null sink works fine.
>
>
> On Wed, Aug 14, 2013 at 12:12 PM, Pankaj Gupta <pa...@brightroll.com>wrote:
>
>> With Null Sink, in the call stack I see a lot of these:
>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>> tid=0x00007feb18001000 nid=0x356b runnable [0x00007feb9e897000]
>>     java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.FileChannelImpl.force0(Native Method)
>>         at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:348)
>>         at
>> org.apache.flume.channel.file.LogFile$Writer.sync(LogFile.java:258)
>>         at
>> org.apache.flume.channel.file.LogFile$Writer.commit(LogFile.java:225)
>>         - locked <0x0000000518c97650> (a
>> org.apache.flume.channel.file.LogFileV3$Writer)
>>         at org.apache.flume.channel.file.Log.commit(Log.java:758)
>>         at org.apache.flume.channel.file.Log.commitTake(Log.java:640)
>>         at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:557)
>>         at
>> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
>>         at org.apache.flume.sink.NullSink.process(NullSink.java:100)
>>         at
>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>         at
>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>         at java.lang.Thread.run(Thread.java:662)
>>
>>
>>
>>
>> On Wed, Aug 14, 2013 at 11:57 AM, Pankaj Gupta <pa...@brightroll.com>wrote:
>>
>>> I tried increasing the dataDirs to 2 and 4 per disk but doesn't seem to
>>> help much. I then replaced the avro sink with null sinks and events are
>>> still filling up in the channel. I tried with both 2 and 4 dataDirs per
>>> disk and null sink, still don't get throughput higher than 1.5 MBps.
>>>
>>>
>>> On Tue, Aug 13, 2013 at 8:30 PM, Brock Noland <br...@cloudera.com>wrote:
>>>
>>>> Increasing the number of file channels will result in more checkpoints.
>>>> Therefore there will be more io than simply increasing the number of
>>>> dataDirs.  However, this might be a case where it'd be nice to relax the
>>>> file channel data consistency constraints a little to get increased
>>>> throughput. That feature does not exist at present.
>>>>
>>>>
>>>> On Tue, Aug 13, 2013 at 10:16 PM, Pankaj Gupta <pa...@brightroll.com>wrote:
>>>>
>>>>> I did try increasing number of FileChannels. At 2 FileChannels per
>>>>> disk performance seemed to be 25% better. At 4 FileChannels per disk
>>>>> performance dropped to even below 1 FileChannel per disk. I will try
>>>>> increasing the dataDirs tomorrow.
>>>>>
>>>>>
>>>>> On Tue, Aug 13, 2013 at 8:06 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>
>>>>>> dataDirs is a comma separated list. Try 3-4 directories and then the
>>>>>> same test.
>>>>>> On Aug 13, 2013 9:58 PM, "Pankaj Gupta" <pa...@brightroll.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Both disks were at around 15-25%.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 13, 2013 at 7:54 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>>>
>>>>>>>> Gotcha. When you run tge test what is tye disk utilization
>>>>>>>> percentage? Iostat can be used for this.
>>>>>>>> On Aug 13, 2013 9:47 PM, "Pankaj Gupta" <pa...@brightroll.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Those are the boxes we want to collect data from. They run flume
>>>>>>>>> and send data through their avro sinks to the avro source on this box. We
>>>>>>>>> are getting data at a pretty good rate and the problem is in fact that the
>>>>>>>>> events don't drain from the FileChannel fast enough and the channel fill
>>>>>>>>> percentage keeps getting higher.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Aug 13, 2013 at 7:41 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>>>>>
>>>>>>>>>> What is sending the events to the avro source?
>>>>>>>>>> On Aug 13, 2013 9:34 PM, "Pankaj Gupta" <pa...@brightroll.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Here's the config:
>>>>>>>>>>> # define channels, one for each disk
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.channels.ch1.type = FILE
>>>>>>>>>>> agent1.channels.ch1.checkpointDir = /flume1/checkpoint
>>>>>>>>>>> agent1.channels.ch1.dataDirs = /flume1/data
>>>>>>>>>>> agent1.channels.ch1.maxFileSize = 375809638400
>>>>>>>>>>> agent1.channels.ch1.capacity = 75000000
>>>>>>>>>>> agent1.channels.ch1.transactionCapacity = 4000
>>>>>>>>>>>
>>>>>>>>>>> agent1.channels.ch2.type = FILE
>>>>>>>>>>> agent1.channels.ch2.checkpointDir = /flume2/checkpoint
>>>>>>>>>>> agent1.channels.ch2.dataDirs = /flume2/data
>>>>>>>>>>> agent1.channels.ch2.maxFileSize = 375809638400
>>>>>>>>>>> agent1.channels.ch2.capacity = 75000000
>>>>>>>>>>> agent1.channels.ch2.transactionCapacity = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> # Define an Avro source named avroSource1
>>>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sources.avroSource1.channels = ch1
>>>>>>>>>>> agent1.sources.avroSource1.type = avro
>>>>>>>>>>> agent1.sources.avroSource1.bind = 0.0.0.0
>>>>>>>>>>> agent1.sources.avroSource1.port = <port>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink1-1-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink1-1-1.channel = ch1
>>>>>>>>>>> agent1.sinks.avroSink1-1-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink1-1-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink1-1-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink1-1-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink1-2-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink1-2-1.channel = ch1
>>>>>>>>>>> agent1.sinks.avroSink1-2-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink1-2-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink1-2-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink1-2-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink1-3-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink1-3-1.channel = ch1
>>>>>>>>>>> agent1.sinks.avroSink1-3-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink1-3-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink1-3-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink1-3-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink1-4-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink1-4-1.channel = ch1
>>>>>>>>>>> agent1.sinks.avroSink1-4-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink1-4-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink1-4-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink1-4-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>>>> which round robin between different hops
>>>>>>>>>>> agent1.sinkgroups.group1.sinks = avroSink1-1-1 avroSink1-2-1
>>>>>>>>>>> avroSink1-3-1 avroSink1-4-1
>>>>>>>>>>> agent1.sinkgroups.group1.processor.type = load_balance
>>>>>>>>>>> agent1.sinkgroups.group1.processor.selector = ROUND_ROBIN
>>>>>>>>>>> agent1.sinkgroups.group1.processor.backoff = true
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> #End of set
>>>>>>>>>>>
>>>>>>>>>>> # Define an Avro source named avroSource2
>>>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sources.avroSource2.channels = ch2
>>>>>>>>>>> agent1.sources.avroSource2.type = avro
>>>>>>>>>>> agent1.sources.avroSource2.bind = 0.0.0.0
>>>>>>>>>>> agent1.sources.avroSource2.port = <port>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink2-1-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink2-1-1.channel = ch2
>>>>>>>>>>> agent1.sinks.avroSink2-1-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink2-1-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink2-1-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink2-1-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink2-2-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink2-2-1.channel = ch2
>>>>>>>>>>> agent1.sinks.avroSink2-2-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink2-2-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink2-2-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink2-2-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink2-3-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink2-3-1.channel = ch2
>>>>>>>>>>> agent1.sinks.avroSink2-3-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink2-3-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink2-3-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink2-3-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink2-4-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink2-4-1.channel = ch2
>>>>>>>>>>> agent1.sinks.avroSink2-4-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink2-4-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink2-4-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink2-4-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>>>> which round robin between different hops
>>>>>>>>>>> agent1.sinkgroups.group2.sinks = avroSink2-1-1 avroSink2-2-1
>>>>>>>>>>> avroSink2-3-1 avroSink2-4-1
>>>>>>>>>>> agent1.sinkgroups.group2.processor.type = load_balance
>>>>>>>>>>> agent1.sinkgroups.group2.processor.selector = ROUND_ROBIN
>>>>>>>>>>> agent1.sinkgroups.group2.processor.backoff = true
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> #End of set
>>>>>>>>>>>
>>>>>>>>>>> # Define an Avro source named avroSource3
>>>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sources.avroSource3.channels = ch1
>>>>>>>>>>> agent1.sources.avroSource3.type = avro
>>>>>>>>>>> agent1.sources.avroSource3.bind = 0.0.0.0
>>>>>>>>>>> agent1.sources.avroSource3.port = <port>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink3-1-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink3-1-1.channel = ch1
>>>>>>>>>>> agent1.sinks.avroSink3-1-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink3-1-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink3-1-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink3-1-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink3-2-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink3-2-1.channel = ch1
>>>>>>>>>>> agent1.sinks.avroSink3-2-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink3-2-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink3-2-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink3-2-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink3-3-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink3-3-1.channel = ch1
>>>>>>>>>>> agent1.sinks.avroSink3-3-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink3-3-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink3-3-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink3-3-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink3-4-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink3-4-1.channel = ch1
>>>>>>>>>>> agent1.sinks.avroSink3-4-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink3-4-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink3-4-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink3-4-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>>>> which round robin between different hops
>>>>>>>>>>> agent1.sinkgroups.group3.sinks = avroSink3-1-1 avroSink3-2-1
>>>>>>>>>>> avroSink3-3-1 avroSink3-4-1
>>>>>>>>>>> agent1.sinkgroups.group3.processor.type = load_balance
>>>>>>>>>>> agent1.sinkgroups.group3.processor.selector = ROUND_ROBIN
>>>>>>>>>>> agent1.sinkgroups.group3.processor.backoff = true
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> #End of set
>>>>>>>>>>>
>>>>>>>>>>> # Define an Avro source named avroSource4
>>>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sources.avroSource4.channels = ch2
>>>>>>>>>>> agent1.sources.avroSource4.type = avro
>>>>>>>>>>> agent1.sources.avroSource4.bind = 0.0.0.0
>>>>>>>>>>> agent1.sources.avroSource4.port = <port>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink4-1-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink4-1-1.channel = ch2
>>>>>>>>>>> agent1.sinks.avroSink4-1-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink4-1-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink4-1-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink4-1-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink4-2-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink4-2-1.channel = ch2
>>>>>>>>>>> agent1.sinks.avroSink4-2-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink4-2-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink4-2-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink4-2-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink4-3-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink4-3-1.channel = ch2
>>>>>>>>>>> agent1.sinks.avroSink4-3-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink4-3-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink4-3-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink4-3-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink4-4-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink4-4-1.channel = ch2
>>>>>>>>>>> agent1.sinks.avroSink4-4-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink4-4-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink4-4-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink4-4-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>>>> which round robin between different hops
>>>>>>>>>>> agent1.sinkgroups.group4.sinks = avroSink4-1-1 avroSink4-2-1
>>>>>>>>>>> avroSink4-3-1 avroSink4-4-1
>>>>>>>>>>> agent1.sinkgroups.group4.processor.type = load_balance
>>>>>>>>>>> agent1.sinkgroups.group4.processor.selector = ROUND_ROBIN
>>>>>>>>>>> agent1.sinkgroups.group4.processor.backoff = true
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> #End of set
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> # Finally, now that we've defined all of our components, tell
>>>>>>>>>>> # agent1 which ones we want to activate.
>>>>>>>>>>> agent1.channels = ch1 ch2
>>>>>>>>>>> agent1.sources = avroSource1 avroSource2 avroSource3 avroSource4
>>>>>>>>>>> agent1.sinks = avroSink1-1-1 avroSink1-2-1 avroSink1-3-1
>>>>>>>>>>> avroSink1-4-1 avroSink2-1-1 avroSink2-2-1 avroSink2-3-1 avroSink2-4-1
>>>>>>>>>>> avroSink3-1-1 avroSink3-2-1 avroSink3-3-1 avroSink3-4-1 avroSink4-1-1
>>>>>>>>>>> avroSink4-2-1 avroSink4-3-1 avroSink4-4-1
>>>>>>>>>>> agent1.sinkgroups = group1 group2 group3 group4
>>>>>>>>>>>
>>>>>>>>>>> There are two disks on the box /flume1 and /flume2
>>>>>>>>>>> 32 cores of CPU, 32 GB RAM, 1GbPS ethernet.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 13, 2013 at 7:22 PM, Brock Noland <
>>>>>>>>>>> brock@cloudera.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Can you share your conf file?
>>>>>>>>>>>> On Aug 13, 2013 9:19 PM, "Hari Shreedharan" <
>>>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>  Even though the writes are done per batch, they don't go to
>>>>>>>>>>>>> disk rightaway - commits are the only ones which actually cause an fsync -
>>>>>>>>>>>>> which is when writes actually go to disk.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Hari
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tuesday, August 13, 2013 at 7:06 PM, Pankaj Gupta wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Looking at the code it seems like the lock and the i/o is done
>>>>>>>>>>>>> per event and not for a batch. Is that correct? If that is the case then it
>>>>>>>>>>>>> seems like there is a lot of overhead per event. The throughput I'm seeing
>>>>>>>>>>>>> is 1 - 1.5 MBps per disk which is way below the sequential read/write
>>>>>>>>>>>>> capacity of the disk which is easily over 50MBps. Adding more sinks doesn't
>>>>>>>>>>>>> help, they just block waiting for the queue to become free. CPU usage is
>>>>>>>>>>>>> 20%, there is enough RAM for page cache so that no read is going to disk.
>>>>>>>>>>>>> The queue seems to be the bottleneck. What is the throughput I should
>>>>>>>>>>>>> expect per disk?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland <
>>>>>>>>>>>>> brock@cloudera.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> The lock is per file. Adding more directories to the channel
>>>>>>>>>>>>> will cause more files to be created. Of course you'll need additional disks
>>>>>>>>>>>>> behind those directories to see any performance increase.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <
>>>>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>  Yes, IO is done inside locks to avoid multiple takes and puts
>>>>>>>>>>>>> getting written out at the same time. Even though Java makes sure the
>>>>>>>>>>>>> writes are serialized, Flume still needs to keep track of some counters
>>>>>>>>>>>>> etc, so the lock is required. Note that the lock you are talking about is
>>>>>>>>>>>>>  in the LogFile class, which represents a single file - so even if the
>>>>>>>>>>>>> write is inside that lock (which is also inside that class itself) that
>>>>>>>>>>>>>  does not cause any contention - because the lock is just preventing two IO
>>>>>>>>>>>>> ops to happen at the same time.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Hari
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> It seems like some i/o is done inside the lock, which means
>>>>>>>>>>>>> that time for taking a lock is proportional to the time for i/o and thus it
>>>>>>>>>>>>> becomes a problem. I apologize in advance if I am wrong but the call stack
>>>>>>>>>>>>> and behavior I'm seeing seems to suggest that. Specifically, it seems that
>>>>>>>>>>>>> we do a write while inside take:
>>>>>>>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>>>>>>>>> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>         at sun.nio.ch.NativeThread.current(Native Method)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>>>>>>>>>>>>>         - locked <0x00000005190ec998> (a java.lang.Object)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>>>>>>>>>>>>>         - locked <0x0000000519111590> (a
>>>>>>>>>>>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>>>>>>>>>>>>         at org.apache.flume.channel.file.Log.take(Log.java:550)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <
>>>>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Since the channel is designed to make sure that events are not
>>>>>>>>>>>>> duplicated to multiple sinks, and to protect against corruption due to
>>>>>>>>>>>>> concurrency issues, we do not need the locking in the channel's flume event
>>>>>>>>>>>>> queue. It is unlikely that locking is what is causing performance issues
>>>>>>>>>>>>> because the channel is heavily I/O bound. If you take a series of thread
>>>>>>>>>>>>> dumps, you will probably see that those threads are moving forward and the
>>>>>>>>>>>>> ones reading/writing from/to disk are the ones which are slower. These
>>>>>>>>>>>>> locks are unlikely to hit performance much.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Hari
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Spent some more time debugging issues with FileChannel. The
>>>>>>>>>>>>> problem seems to lock contention reading from FlumeEventQueue:
>>>>>>>>>>>>>
>>>>>>>>>>>>> I see a lot of threads like this:
>>>>>>>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>>>>>>>>> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
>>>>>>>>>>>>> [0x00007f821afee000]
>>>>>>>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>>>>>>>>>>>>>         - waiting to lock <0x0000000518ee4c90> (a
>>>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have two file channels and 8 Avro Sinks per file channel. I
>>>>>>>>>>>>> added more sinks because they weren't draining fast enough. It seems like
>>>>>>>>>>>>> they send the batch then wait for an ack before sending again, thus sends
>>>>>>>>>>>>> are not pipelined and having more sinks seemed like a good way of getting
>>>>>>>>>>>>> some parallelism.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Here's the full stack trace:
>>>>>>>>>>>>> 2013-08-13 15:30:32
>>>>>>>>>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02
>>>>>>>>>>>>> mixed mode):
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
>>>>>>>>>>>>>    java.lang.Thread.State: WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x000000056d932120> (a
>>>>>>>>>>>>> java.util.concurrent.CountDownLatch$Sync)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
>>>>>>>>>>>>> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
>>>>>>>>>>>>>  at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
>>>>>>>>>>>>> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
>>>>>>>>>>>>> at $Proxy7.appendBatch(Unknown Source)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000
>>>>>>>>>>>>> nid=0x41a1 waiting on condition [0x0000000000000000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x00000005194c3340> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x00000005191a16d8> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x0000000519174e08> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x0000000519174e98> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x00000005195f9670> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x0000000519174ce8> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x00000005191a1918> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x00000005191e7f60> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153
>>>>>>>>>>>>> runnable [0x00007f8210e4d000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x000000054a2917a8> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x000000054a2c5348> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x0000000519174f58> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146
>>>>>>>>>>>>> runnable [0x00007f8210f4e000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x0000000546e5dd28> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x0000000546ec1ea0> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118
>>>>>>>>>>>>> runnable [0x00007f821104f000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x0000000544f763c0> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x0000000544ee55f8> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111
>>>>>>>>>>>>> runnable [0x00007f8211150000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x000000054473d5a8> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x000000054474d088> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110
>>>>>>>>>>>>> runnable [0x00007f8211251000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x00000005437f4f70> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x0000000543803478> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f
>>>>>>>>>>>>> runnable [0x00007f8211352000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x00000005422a9018> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x00000005422ca310> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e
>>>>>>>>>>>>> runnable [0x00007f8211453000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x0000000541063d50> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x00000005410c8240> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d
>>>>>>>>>>>>> runnable [0x00007f8211554000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x0000000540dc1110> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x0000000540c94928> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a
>>>>>>>>>>>>> runnable [0x00007f8211655000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x000000053fbbc3b0> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x000000053f897d00> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109
>>>>>>>>>>>>> runnable [0x00007f8211756000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x000000053f812860> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x000000053f897cb8> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd
>>>>>>>>>>>>> runnable [0x00007f8211857000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x000000053e6e5570> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x000000053e714e70> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x0000000519175138> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x0000000519174c28> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x00000005191751c8> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0
>>>>>>>>>>>>> runnable [0x00007f8211c5b000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x0000000536579db0> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x0000000536585830> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x00000005191750a8> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1
>>>>>>>>>>>>> runnable [0x00007f8211e5d000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x00000005313943a8> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x00000005314e0fe0> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x000000051936f460> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x000000051936dfa8> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de
>>>>>>>>>>>>> runnable [0x00007f8212160000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x000000052dad4468> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x000000052da31528> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd
>>>>>>>>>>>>> runnable [0x00007f8212261000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x000000052c73c5f0> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x000000052c652608> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc
>>>>>>>>>>>>> runnable [0x00007f8212362000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x000000052b8da998> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x000000052b8fc590> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db
>>>>>>>>>>>>> waiting for monitor entry [0x00007f8212462000]
>>>>>>>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>>>>>>>>>>>>> - waiting to lock <0x0000000518ee4c90> (a
>>>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>>>>>>>>>>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>>>>>>>>>>>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x000000051936df18> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x0000000518fe7ae8> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8
>>>>>>>>>>>>> runnable [0x00007f8212766000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x00000005263d6ba8> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x00000005264b1a48> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7
>>>>>>>>>>>>> runnable [0x00007f8212867000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x0000000525d1d0e0> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x0000000525d2cdf8> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6
>>>>>>>>>>>>> runnable [0x00007f8212968000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x0000000525a689d8> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x0000000525a24570> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5
>>>>>>>>>>>>> runnable [0x00007f8212a69000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x0000000524e1e7d8> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x0000000524cfd670> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4
>>>>>>>>>>>>> runnable [0x00007f8212b6a000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x0000000524d437a8> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x0000000524ab3330> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>
>>>>>>>>>>>>> ...
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 |
>>>>> pankaj@brightroll.com
>>>>>
>>>>> Pankaj Gupta | Software Engineer
>>>>>
>>>>> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>>>>>
>>>>>
>>>>> United States | Canada | United Kingdom | Germany
>>>>>
>>>>>
>>>>> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
>>>>> !
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>>>>
>>>
>>>
>>>
>>> --
>>>
>>>
>>> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 |
>>> pankaj@brightroll.com
>>>
>>> Pankaj Gupta | Software Engineer
>>>
>>> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>>>
>>>
>>> United States | Canada | United Kingdom | Germany
>>>
>>>
>>> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
>>> !
>>>
>>
>>
>>
>> --
>>
>>
>> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 |
>> pankaj@brightroll.com
>>
>> Pankaj Gupta | Software Engineer
>>
>> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>>
>>
>> United States | Canada | United Kingdom | Germany
>>
>>
>> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
>> !
>>
>
>
>
> --
>
>
> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com
>
> Pankaj Gupta | Software Engineer
>
> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>
>
> United States | Canada | United Kingdom | Germany
>
>
> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
> !
>



-- 


*P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com

Pankaj Gupta | Software Engineer

*BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com


United States | Canada | United Kingdom | Germany


We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
!

Re: Lock contention in FileChannel

[Hari Shreedharan <hs...@cloudera.com>]

You can increase the number of Avro Sinks active at a time, by increasing
the number of sink groups. This should give better throughput.


Hari


On Wed, Aug 14, 2013 at 12:34 PM, Pankaj Gupta <pa...@brightroll.com>wrote:

> Null sink is actually holding up. I hadn't specified the batchSize so the
> default of 100 was being picked up. When I set batchSize to 4000 it started
> consuming all events and the channels are not filling up. So it seems that
> the problem is a combination of AvroSink with FileChannel. Memory Channel
> with Avro Sink works fine and FileChannel with null sink works fine.
>
>
> On Wed, Aug 14, 2013 at 12:12 PM, Pankaj Gupta <pa...@brightroll.com>wrote:
>
>> With Null Sink, in the call stack I see a lot of these:
>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>> tid=0x00007feb18001000 nid=0x356b runnable [0x00007feb9e897000]
>>     java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.FileChannelImpl.force0(Native Method)
>>         at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:348)
>>         at
>> org.apache.flume.channel.file.LogFile$Writer.sync(LogFile.java:258)
>>         at
>> org.apache.flume.channel.file.LogFile$Writer.commit(LogFile.java:225)
>>         - locked <0x0000000518c97650> (a
>> org.apache.flume.channel.file.LogFileV3$Writer)
>>         at org.apache.flume.channel.file.Log.commit(Log.java:758)
>>         at org.apache.flume.channel.file.Log.commitTake(Log.java:640)
>>         at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:557)
>>         at
>> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
>>         at org.apache.flume.sink.NullSink.process(NullSink.java:100)
>>         at
>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>         at
>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>         at java.lang.Thread.run(Thread.java:662)
>>
>>
>>
>>
>> On Wed, Aug 14, 2013 at 11:57 AM, Pankaj Gupta <pa...@brightroll.com>wrote:
>>
>>> I tried increasing the dataDirs to 2 and 4 per disk but doesn't seem to
>>> help much. I then replaced the avro sink with null sinks and events are
>>> still filling up in the channel. I tried with both 2 and 4 dataDirs per
>>> disk and null sink, still don't get throughput higher than 1.5 MBps.
>>>
>>>
>>> On Tue, Aug 13, 2013 at 8:30 PM, Brock Noland <br...@cloudera.com>wrote:
>>>
>>>> Increasing the number of file channels will result in more checkpoints.
>>>> Therefore there will be more io than simply increasing the number of
>>>> dataDirs.  However, this might be a case where it'd be nice to relax the
>>>> file channel data consistency constraints a little to get increased
>>>> throughput. That feature does not exist at present.
>>>>
>>>>
>>>> On Tue, Aug 13, 2013 at 10:16 PM, Pankaj Gupta <pa...@brightroll.com>wrote:
>>>>
>>>>> I did try increasing number of FileChannels. At 2 FileChannels per
>>>>> disk performance seemed to be 25% better. At 4 FileChannels per disk
>>>>> performance dropped to even below 1 FileChannel per disk. I will try
>>>>> increasing the dataDirs tomorrow.
>>>>>
>>>>>
>>>>> On Tue, Aug 13, 2013 at 8:06 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>
>>>>>> dataDirs is a comma separated list. Try 3-4 directories and then the
>>>>>> same test.
>>>>>> On Aug 13, 2013 9:58 PM, "Pankaj Gupta" <pa...@brightroll.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Both disks were at around 15-25%.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 13, 2013 at 7:54 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>>>
>>>>>>>> Gotcha. When you run tge test what is tye disk utilization
>>>>>>>> percentage? Iostat can be used for this.
>>>>>>>> On Aug 13, 2013 9:47 PM, "Pankaj Gupta" <pa...@brightroll.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Those are the boxes we want to collect data from. They run flume
>>>>>>>>> and send data through their avro sinks to the avro source on this box. We
>>>>>>>>> are getting data at a pretty good rate and the problem is in fact that the
>>>>>>>>> events don't drain from the FileChannel fast enough and the channel fill
>>>>>>>>> percentage keeps getting higher.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Aug 13, 2013 at 7:41 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>>>>>
>>>>>>>>>> What is sending the events to the avro source?
>>>>>>>>>> On Aug 13, 2013 9:34 PM, "Pankaj Gupta" <pa...@brightroll.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Here's the config:
>>>>>>>>>>> # define channels, one for each disk
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.channels.ch1.type = FILE
>>>>>>>>>>> agent1.channels.ch1.checkpointDir = /flume1/checkpoint
>>>>>>>>>>> agent1.channels.ch1.dataDirs = /flume1/data
>>>>>>>>>>> agent1.channels.ch1.maxFileSize = 375809638400
>>>>>>>>>>> agent1.channels.ch1.capacity = 75000000
>>>>>>>>>>> agent1.channels.ch1.transactionCapacity = 4000
>>>>>>>>>>>
>>>>>>>>>>> agent1.channels.ch2.type = FILE
>>>>>>>>>>> agent1.channels.ch2.checkpointDir = /flume2/checkpoint
>>>>>>>>>>> agent1.channels.ch2.dataDirs = /flume2/data
>>>>>>>>>>> agent1.channels.ch2.maxFileSize = 375809638400
>>>>>>>>>>> agent1.channels.ch2.capacity = 75000000
>>>>>>>>>>> agent1.channels.ch2.transactionCapacity = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> # Define an Avro source named avroSource1
>>>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sources.avroSource1.channels = ch1
>>>>>>>>>>> agent1.sources.avroSource1.type = avro
>>>>>>>>>>> agent1.sources.avroSource1.bind = 0.0.0.0
>>>>>>>>>>> agent1.sources.avroSource1.port = <port>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink1-1-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink1-1-1.channel = ch1
>>>>>>>>>>> agent1.sinks.avroSink1-1-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink1-1-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink1-1-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink1-1-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink1-2-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink1-2-1.channel = ch1
>>>>>>>>>>> agent1.sinks.avroSink1-2-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink1-2-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink1-2-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink1-2-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink1-3-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink1-3-1.channel = ch1
>>>>>>>>>>> agent1.sinks.avroSink1-3-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink1-3-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink1-3-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink1-3-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink1-4-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink1-4-1.channel = ch1
>>>>>>>>>>> agent1.sinks.avroSink1-4-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink1-4-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink1-4-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink1-4-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>>>> which round robin between different hops
>>>>>>>>>>> agent1.sinkgroups.group1.sinks = avroSink1-1-1 avroSink1-2-1
>>>>>>>>>>> avroSink1-3-1 avroSink1-4-1
>>>>>>>>>>> agent1.sinkgroups.group1.processor.type = load_balance
>>>>>>>>>>> agent1.sinkgroups.group1.processor.selector = ROUND_ROBIN
>>>>>>>>>>> agent1.sinkgroups.group1.processor.backoff = true
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> #End of set
>>>>>>>>>>>
>>>>>>>>>>> # Define an Avro source named avroSource2
>>>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sources.avroSource2.channels = ch2
>>>>>>>>>>> agent1.sources.avroSource2.type = avro
>>>>>>>>>>> agent1.sources.avroSource2.bind = 0.0.0.0
>>>>>>>>>>> agent1.sources.avroSource2.port = <port>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink2-1-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink2-1-1.channel = ch2
>>>>>>>>>>> agent1.sinks.avroSink2-1-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink2-1-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink2-1-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink2-1-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink2-2-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink2-2-1.channel = ch2
>>>>>>>>>>> agent1.sinks.avroSink2-2-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink2-2-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink2-2-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink2-2-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink2-3-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink2-3-1.channel = ch2
>>>>>>>>>>> agent1.sinks.avroSink2-3-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink2-3-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink2-3-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink2-3-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink2-4-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink2-4-1.channel = ch2
>>>>>>>>>>> agent1.sinks.avroSink2-4-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink2-4-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink2-4-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink2-4-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>>>> which round robin between different hops
>>>>>>>>>>> agent1.sinkgroups.group2.sinks = avroSink2-1-1 avroSink2-2-1
>>>>>>>>>>> avroSink2-3-1 avroSink2-4-1
>>>>>>>>>>> agent1.sinkgroups.group2.processor.type = load_balance
>>>>>>>>>>> agent1.sinkgroups.group2.processor.selector = ROUND_ROBIN
>>>>>>>>>>> agent1.sinkgroups.group2.processor.backoff = true
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> #End of set
>>>>>>>>>>>
>>>>>>>>>>> # Define an Avro source named avroSource3
>>>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sources.avroSource3.channels = ch1
>>>>>>>>>>> agent1.sources.avroSource3.type = avro
>>>>>>>>>>> agent1.sources.avroSource3.bind = 0.0.0.0
>>>>>>>>>>> agent1.sources.avroSource3.port = <port>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink3-1-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink3-1-1.channel = ch1
>>>>>>>>>>> agent1.sinks.avroSink3-1-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink3-1-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink3-1-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink3-1-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink3-2-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink3-2-1.channel = ch1
>>>>>>>>>>> agent1.sinks.avroSink3-2-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink3-2-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink3-2-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink3-2-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink3-3-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink3-3-1.channel = ch1
>>>>>>>>>>> agent1.sinks.avroSink3-3-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink3-3-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink3-3-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink3-3-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink3-4-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink3-4-1.channel = ch1
>>>>>>>>>>> agent1.sinks.avroSink3-4-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink3-4-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink3-4-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink3-4-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>>>> which round robin between different hops
>>>>>>>>>>> agent1.sinkgroups.group3.sinks = avroSink3-1-1 avroSink3-2-1
>>>>>>>>>>> avroSink3-3-1 avroSink3-4-1
>>>>>>>>>>> agent1.sinkgroups.group3.processor.type = load_balance
>>>>>>>>>>> agent1.sinkgroups.group3.processor.selector = ROUND_ROBIN
>>>>>>>>>>> agent1.sinkgroups.group3.processor.backoff = true
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> #End of set
>>>>>>>>>>>
>>>>>>>>>>> # Define an Avro source named avroSource4
>>>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sources.avroSource4.channels = ch2
>>>>>>>>>>> agent1.sources.avroSource4.type = avro
>>>>>>>>>>> agent1.sources.avroSource4.bind = 0.0.0.0
>>>>>>>>>>> agent1.sources.avroSource4.port = <port>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink4-1-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink4-1-1.channel = ch2
>>>>>>>>>>> agent1.sinks.avroSink4-1-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink4-1-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink4-1-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink4-1-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink4-2-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink4-2-1.channel = ch2
>>>>>>>>>>> agent1.sinks.avroSink4-2-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink4-2-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink4-2-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink4-2-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink4-3-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink4-3-1.channel = ch2
>>>>>>>>>>> agent1.sinks.avroSink4-3-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink4-3-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink4-3-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink4-3-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> agent1.sinks.avroSink4-4-1.type = avro
>>>>>>>>>>> agent1.sinks.avroSink4-4-1.channel = ch2
>>>>>>>>>>> agent1.sinks.avroSink4-4-1.hostname = <hostname>
>>>>>>>>>>> agent1.sinks.avroSink4-4-1.port = <port>
>>>>>>>>>>> agent1.sinks.avroSink4-4-1.connect-timeout = 300000
>>>>>>>>>>> agent1.sinks.avroSink4-4-1.batch-size = 4000
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>>>> which round robin between different hops
>>>>>>>>>>> agent1.sinkgroups.group4.sinks = avroSink4-1-1 avroSink4-2-1
>>>>>>>>>>> avroSink4-3-1 avroSink4-4-1
>>>>>>>>>>> agent1.sinkgroups.group4.processor.type = load_balance
>>>>>>>>>>> agent1.sinkgroups.group4.processor.selector = ROUND_ROBIN
>>>>>>>>>>> agent1.sinkgroups.group4.processor.backoff = true
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> #End of set
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> # Finally, now that we've defined all of our components, tell
>>>>>>>>>>> # agent1 which ones we want to activate.
>>>>>>>>>>> agent1.channels = ch1 ch2
>>>>>>>>>>> agent1.sources = avroSource1 avroSource2 avroSource3 avroSource4
>>>>>>>>>>> agent1.sinks = avroSink1-1-1 avroSink1-2-1 avroSink1-3-1
>>>>>>>>>>> avroSink1-4-1 avroSink2-1-1 avroSink2-2-1 avroSink2-3-1 avroSink2-4-1
>>>>>>>>>>> avroSink3-1-1 avroSink3-2-1 avroSink3-3-1 avroSink3-4-1 avroSink4-1-1
>>>>>>>>>>> avroSink4-2-1 avroSink4-3-1 avroSink4-4-1
>>>>>>>>>>> agent1.sinkgroups = group1 group2 group3 group4
>>>>>>>>>>>
>>>>>>>>>>> There are two disks on the box /flume1 and /flume2
>>>>>>>>>>> 32 cores of CPU, 32 GB RAM, 1GbPS ethernet.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 13, 2013 at 7:22 PM, Brock Noland <
>>>>>>>>>>> brock@cloudera.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Can you share your conf file?
>>>>>>>>>>>> On Aug 13, 2013 9:19 PM, "Hari Shreedharan" <
>>>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>  Even though the writes are done per batch, they don't go to
>>>>>>>>>>>>> disk rightaway - commits are the only ones which actually cause an fsync -
>>>>>>>>>>>>> which is when writes actually go to disk.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Hari
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tuesday, August 13, 2013 at 7:06 PM, Pankaj Gupta wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Looking at the code it seems like the lock and the i/o is done
>>>>>>>>>>>>> per event and not for a batch. Is that correct? If that is the case then it
>>>>>>>>>>>>> seems like there is a lot of overhead per event. The throughput I'm seeing
>>>>>>>>>>>>> is 1 - 1.5 MBps per disk which is way below the sequential read/write
>>>>>>>>>>>>> capacity of the disk which is easily over 50MBps. Adding more sinks doesn't
>>>>>>>>>>>>> help, they just block waiting for the queue to become free. CPU usage is
>>>>>>>>>>>>> 20%, there is enough RAM for page cache so that no read is going to disk.
>>>>>>>>>>>>> The queue seems to be the bottleneck. What is the throughput I should
>>>>>>>>>>>>> expect per disk?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland <
>>>>>>>>>>>>> brock@cloudera.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> The lock is per file. Adding more directories to the channel
>>>>>>>>>>>>> will cause more files to be created. Of course you'll need additional disks
>>>>>>>>>>>>> behind those directories to see any performance increase.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <
>>>>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>  Yes, IO is done inside locks to avoid multiple takes and puts
>>>>>>>>>>>>> getting written out at the same time. Even though Java makes sure the
>>>>>>>>>>>>> writes are serialized, Flume still needs to keep track of some counters
>>>>>>>>>>>>> etc, so the lock is required. Note that the lock you are talking about is
>>>>>>>>>>>>>  in the LogFile class, which represents a single file - so even if the
>>>>>>>>>>>>> write is inside that lock (which is also inside that class itself) that
>>>>>>>>>>>>>  does not cause any contention - because the lock is just preventing two IO
>>>>>>>>>>>>> ops to happen at the same time.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Hari
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> It seems like some i/o is done inside the lock, which means
>>>>>>>>>>>>> that time for taking a lock is proportional to the time for i/o and thus it
>>>>>>>>>>>>> becomes a problem. I apologize in advance if I am wrong but the call stack
>>>>>>>>>>>>> and behavior I'm seeing seems to suggest that. Specifically, it seems that
>>>>>>>>>>>>> we do a write while inside take:
>>>>>>>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>>>>>>>>> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>         at sun.nio.ch.NativeThread.current(Native Method)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>>>>>>>>>>>>>         - locked <0x00000005190ec998> (a java.lang.Object)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>>>>>>>>>>>>>         - locked <0x0000000519111590> (a
>>>>>>>>>>>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>>>>>>>>>>>>         at org.apache.flume.channel.file.Log.take(Log.java:550)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <
>>>>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Since the channel is designed to make sure that events are not
>>>>>>>>>>>>> duplicated to multiple sinks, and to protect against corruption due to
>>>>>>>>>>>>> concurrency issues, we do not need the locking in the channel's flume event
>>>>>>>>>>>>> queue. It is unlikely that locking is what is causing performance issues
>>>>>>>>>>>>> because the channel is heavily I/O bound. If you take a series of thread
>>>>>>>>>>>>> dumps, you will probably see that those threads are moving forward and the
>>>>>>>>>>>>> ones reading/writing from/to disk are the ones which are slower. These
>>>>>>>>>>>>> locks are unlikely to hit performance much.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Hari
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Spent some more time debugging issues with FileChannel. The
>>>>>>>>>>>>> problem seems to lock contention reading from FlumeEventQueue:
>>>>>>>>>>>>>
>>>>>>>>>>>>> I see a lot of threads like this:
>>>>>>>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>>>>>>>>> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
>>>>>>>>>>>>> [0x00007f821afee000]
>>>>>>>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>>>>>>>>>>>>>         - waiting to lock <0x0000000518ee4c90> (a
>>>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have two file channels and 8 Avro Sinks per file channel. I
>>>>>>>>>>>>> added more sinks because they weren't draining fast enough. It seems like
>>>>>>>>>>>>> they send the batch then wait for an ack before sending again, thus sends
>>>>>>>>>>>>> are not pipelined and having more sinks seemed like a good way of getting
>>>>>>>>>>>>> some parallelism.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Here's the full stack trace:
>>>>>>>>>>>>> 2013-08-13 15:30:32
>>>>>>>>>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02
>>>>>>>>>>>>> mixed mode):
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
>>>>>>>>>>>>>    java.lang.Thread.State: WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x000000056d932120> (a
>>>>>>>>>>>>> java.util.concurrent.CountDownLatch$Sync)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
>>>>>>>>>>>>> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
>>>>>>>>>>>>>  at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
>>>>>>>>>>>>> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
>>>>>>>>>>>>> at $Proxy7.appendBatch(Unknown Source)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000
>>>>>>>>>>>>> nid=0x41a1 waiting on condition [0x0000000000000000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x00000005194c3340> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x00000005191a16d8> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x0000000519174e08> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x0000000519174e98> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x00000005195f9670> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x0000000519174ce8> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x00000005191a1918> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x00000005191e7f60> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153
>>>>>>>>>>>>> runnable [0x00007f8210e4d000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x000000054a2917a8> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x000000054a2c5348> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x0000000519174f58> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146
>>>>>>>>>>>>> runnable [0x00007f8210f4e000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x0000000546e5dd28> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x0000000546ec1ea0> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118
>>>>>>>>>>>>> runnable [0x00007f821104f000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x0000000544f763c0> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x0000000544ee55f8> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111
>>>>>>>>>>>>> runnable [0x00007f8211150000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x000000054473d5a8> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x000000054474d088> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110
>>>>>>>>>>>>> runnable [0x00007f8211251000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x00000005437f4f70> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x0000000543803478> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f
>>>>>>>>>>>>> runnable [0x00007f8211352000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x00000005422a9018> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x00000005422ca310> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e
>>>>>>>>>>>>> runnable [0x00007f8211453000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x0000000541063d50> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x00000005410c8240> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d
>>>>>>>>>>>>> runnable [0x00007f8211554000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x0000000540dc1110> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x0000000540c94928> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a
>>>>>>>>>>>>> runnable [0x00007f8211655000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x000000053fbbc3b0> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x000000053f897d00> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109
>>>>>>>>>>>>> runnable [0x00007f8211756000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x000000053f812860> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x000000053f897cb8> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd
>>>>>>>>>>>>> runnable [0x00007f8211857000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x000000053e6e5570> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x000000053e714e70> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x0000000519175138> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x0000000519174c28> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x00000005191751c8> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0
>>>>>>>>>>>>> runnable [0x00007f8211c5b000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x0000000536579db0> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x0000000536585830> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x00000005191750a8> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1
>>>>>>>>>>>>> runnable [0x00007f8211e5d000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x00000005313943a8> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x00000005314e0fe0> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x000000051936f460> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x000000051936dfa8> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de
>>>>>>>>>>>>> runnable [0x00007f8212160000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x000000052dad4468> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x000000052da31528> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd
>>>>>>>>>>>>> runnable [0x00007f8212261000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x000000052c73c5f0> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x000000052c652608> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc
>>>>>>>>>>>>> runnable [0x00007f8212362000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x000000052b8da998> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x000000052b8fc590> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db
>>>>>>>>>>>>> waiting for monitor entry [0x00007f8212462000]
>>>>>>>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>>>>>>>>>>>>> - waiting to lock <0x0000000518ee4c90> (a
>>>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>>>>>>>>>>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>>>>>>>>>>>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x000000051936df18> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>>> tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000]
>>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>>> - parking to wait for  <0x0000000518fe7ae8> (a
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8
>>>>>>>>>>>>> runnable [0x00007f8212766000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x00000005263d6ba8> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x00000005264b1a48> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7
>>>>>>>>>>>>> runnable [0x00007f8212867000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x0000000525d1d0e0> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x0000000525d2cdf8> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6
>>>>>>>>>>>>> runnable [0x00007f8212968000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x0000000525a689d8> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x0000000525a24570> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5
>>>>>>>>>>>>> runnable [0x00007f8212a69000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x0000000524e1e7d8> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x0000000524cfd670> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>
>>>>>>>>>>>>> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4
>>>>>>>>>>>>> runnable [0x00007f8212b6a000]
>>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>>  - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>>> - locked <0x0000000524d437a8> (a
>>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>>  - locked <0x0000000524ab3330> (a
>>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>>  at
>>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>>
>>>>>>>>>>>>> ...
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 |
>>>>> pankaj@brightroll.com
>>>>>
>>>>> Pankaj Gupta | Software Engineer
>>>>>
>>>>> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>>>>>
>>>>>
>>>>> United States | Canada | United Kingdom | Germany
>>>>>
>>>>>
>>>>> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
>>>>> !
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>>>>
>>>
>>>
>>>
>>> --
>>>
>>>
>>> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 |
>>> pankaj@brightroll.com
>>>
>>> Pankaj Gupta | Software Engineer
>>>
>>> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>>>
>>>
>>> United States | Canada | United Kingdom | Germany
>>>
>>>
>>> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
>>> !
>>>
>>
>>
>>
>> --
>>
>>
>> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 |
>> pankaj@brightroll.com
>>
>> Pankaj Gupta | Software Engineer
>>
>> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>>
>>
>> United States | Canada | United Kingdom | Germany
>>
>>
>> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
>> !
>>
>
>
>
> --
>
>
> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com
>
> Pankaj Gupta | Software Engineer
>
> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>
>
> United States | Canada | United Kingdom | Germany
>
>
> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
> !
>

Re: Lock contention in FileChannel

[Pankaj Gupta <pa...@brightroll.com>]

Null sink is actually holding up. I hadn't specified the batchSize so the
default of 100 was being picked up. When I set batchSize to 4000 it started
consuming all events and the channels are not filling up. So it seems that
the problem is a combination of AvroSink with FileChannel. Memory Channel
with Avro Sink works fine and FileChannel with null sink works fine.


On Wed, Aug 14, 2013 at 12:12 PM, Pankaj Gupta <pa...@brightroll.com>wrote:

> With Null Sink, in the call stack I see a lot of these:
> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
> tid=0x00007feb18001000 nid=0x356b runnable [0x00007feb9e897000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.FileChannelImpl.force0(Native Method)
>         at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:348)
>         at
> org.apache.flume.channel.file.LogFile$Writer.sync(LogFile.java:258)
>         at
> org.apache.flume.channel.file.LogFile$Writer.commit(LogFile.java:225)
>         - locked <0x0000000518c97650> (a
> org.apache.flume.channel.file.LogFileV3$Writer)
>         at org.apache.flume.channel.file.Log.commit(Log.java:758)
>         at org.apache.flume.channel.file.Log.commitTake(Log.java:640)
>         at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:557)
>         at
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
>         at org.apache.flume.sink.NullSink.process(NullSink.java:100)
>         at
> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:662)
>
>
>
>
> On Wed, Aug 14, 2013 at 11:57 AM, Pankaj Gupta <pa...@brightroll.com>wrote:
>
>> I tried increasing the dataDirs to 2 and 4 per disk but doesn't seem to
>> help much. I then replaced the avro sink with null sinks and events are
>> still filling up in the channel. I tried with both 2 and 4 dataDirs per
>> disk and null sink, still don't get throughput higher than 1.5 MBps.
>>
>>
>> On Tue, Aug 13, 2013 at 8:30 PM, Brock Noland <br...@cloudera.com> wrote:
>>
>>> Increasing the number of file channels will result in more checkpoints.
>>> Therefore there will be more io than simply increasing the number of
>>> dataDirs.  However, this might be a case where it'd be nice to relax the
>>> file channel data consistency constraints a little to get increased
>>> throughput. That feature does not exist at present.
>>>
>>>
>>> On Tue, Aug 13, 2013 at 10:16 PM, Pankaj Gupta <pa...@brightroll.com>wrote:
>>>
>>>> I did try increasing number of FileChannels. At 2 FileChannels per disk
>>>> performance seemed to be 25% better. At 4 FileChannels per disk performance
>>>> dropped to even below 1 FileChannel per disk. I will try increasing the
>>>> dataDirs tomorrow.
>>>>
>>>>
>>>> On Tue, Aug 13, 2013 at 8:06 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>
>>>>> dataDirs is a comma separated list. Try 3-4 directories and then the
>>>>> same test.
>>>>> On Aug 13, 2013 9:58 PM, "Pankaj Gupta" <pa...@brightroll.com> wrote:
>>>>>
>>>>>> Both disks were at around 15-25%.
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 13, 2013 at 7:54 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>>
>>>>>>> Gotcha. When you run tge test what is tye disk utilization
>>>>>>> percentage? Iostat can be used for this.
>>>>>>> On Aug 13, 2013 9:47 PM, "Pankaj Gupta" <pa...@brightroll.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Those are the boxes we want to collect data from. They run flume
>>>>>>>> and send data through their avro sinks to the avro source on this box. We
>>>>>>>> are getting data at a pretty good rate and the problem is in fact that the
>>>>>>>> events don't drain from the FileChannel fast enough and the channel fill
>>>>>>>> percentage keeps getting higher.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Aug 13, 2013 at 7:41 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>>>>
>>>>>>>>> What is sending the events to the avro source?
>>>>>>>>> On Aug 13, 2013 9:34 PM, "Pankaj Gupta" <pa...@brightroll.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Here's the config:
>>>>>>>>>> # define channels, one for each disk
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> agent1.channels.ch1.type = FILE
>>>>>>>>>> agent1.channels.ch1.checkpointDir = /flume1/checkpoint
>>>>>>>>>> agent1.channels.ch1.dataDirs = /flume1/data
>>>>>>>>>> agent1.channels.ch1.maxFileSize = 375809638400
>>>>>>>>>> agent1.channels.ch1.capacity = 75000000
>>>>>>>>>> agent1.channels.ch1.transactionCapacity = 4000
>>>>>>>>>>
>>>>>>>>>> agent1.channels.ch2.type = FILE
>>>>>>>>>> agent1.channels.ch2.checkpointDir = /flume2/checkpoint
>>>>>>>>>> agent1.channels.ch2.dataDirs = /flume2/data
>>>>>>>>>> agent1.channels.ch2.maxFileSize = 375809638400
>>>>>>>>>> agent1.channels.ch2.capacity = 75000000
>>>>>>>>>> agent1.channels.ch2.transactionCapacity = 4000
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> # Define an Avro source named avroSource1
>>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> agent1.sources.avroSource1.channels = ch1
>>>>>>>>>> agent1.sources.avroSource1.type = avro
>>>>>>>>>> agent1.sources.avroSource1.bind = 0.0.0.0
>>>>>>>>>> agent1.sources.avroSource1.port = <port>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> agent1.sinks.avroSink1-1-1.type = avro
>>>>>>>>>> agent1.sinks.avroSink1-1-1.channel = ch1
>>>>>>>>>> agent1.sinks.avroSink1-1-1.hostname = <hostname>
>>>>>>>>>> agent1.sinks.avroSink1-1-1.port = <port>
>>>>>>>>>> agent1.sinks.avroSink1-1-1.connect-timeout = 300000
>>>>>>>>>> agent1.sinks.avroSink1-1-1.batch-size = 4000
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> agent1.sinks.avroSink1-2-1.type = avro
>>>>>>>>>> agent1.sinks.avroSink1-2-1.channel = ch1
>>>>>>>>>> agent1.sinks.avroSink1-2-1.hostname = <hostname>
>>>>>>>>>> agent1.sinks.avroSink1-2-1.port = <port>
>>>>>>>>>> agent1.sinks.avroSink1-2-1.connect-timeout = 300000
>>>>>>>>>> agent1.sinks.avroSink1-2-1.batch-size = 4000
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> agent1.sinks.avroSink1-3-1.type = avro
>>>>>>>>>> agent1.sinks.avroSink1-3-1.channel = ch1
>>>>>>>>>> agent1.sinks.avroSink1-3-1.hostname = <hostname>
>>>>>>>>>> agent1.sinks.avroSink1-3-1.port = <port>
>>>>>>>>>> agent1.sinks.avroSink1-3-1.connect-timeout = 300000
>>>>>>>>>> agent1.sinks.avroSink1-3-1.batch-size = 4000
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> agent1.sinks.avroSink1-4-1.type = avro
>>>>>>>>>> agent1.sinks.avroSink1-4-1.channel = ch1
>>>>>>>>>> agent1.sinks.avroSink1-4-1.hostname = <hostname>
>>>>>>>>>> agent1.sinks.avroSink1-4-1.port = <port>
>>>>>>>>>> agent1.sinks.avroSink1-4-1.connect-timeout = 300000
>>>>>>>>>> agent1.sinks.avroSink1-4-1.batch-size = 4000
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>>> which round robin between different hops
>>>>>>>>>> agent1.sinkgroups.group1.sinks = avroSink1-1-1 avroSink1-2-1
>>>>>>>>>> avroSink1-3-1 avroSink1-4-1
>>>>>>>>>> agent1.sinkgroups.group1.processor.type = load_balance
>>>>>>>>>> agent1.sinkgroups.group1.processor.selector = ROUND_ROBIN
>>>>>>>>>> agent1.sinkgroups.group1.processor.backoff = true
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> #End of set
>>>>>>>>>>
>>>>>>>>>> # Define an Avro source named avroSource2
>>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> agent1.sources.avroSource2.channels = ch2
>>>>>>>>>> agent1.sources.avroSource2.type = avro
>>>>>>>>>> agent1.sources.avroSource2.bind = 0.0.0.0
>>>>>>>>>> agent1.sources.avroSource2.port = <port>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> agent1.sinks.avroSink2-1-1.type = avro
>>>>>>>>>> agent1.sinks.avroSink2-1-1.channel = ch2
>>>>>>>>>> agent1.sinks.avroSink2-1-1.hostname = <hostname>
>>>>>>>>>> agent1.sinks.avroSink2-1-1.port = <port>
>>>>>>>>>> agent1.sinks.avroSink2-1-1.connect-timeout = 300000
>>>>>>>>>> agent1.sinks.avroSink2-1-1.batch-size = 4000
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> agent1.sinks.avroSink2-2-1.type = avro
>>>>>>>>>> agent1.sinks.avroSink2-2-1.channel = ch2
>>>>>>>>>> agent1.sinks.avroSink2-2-1.hostname = <hostname>
>>>>>>>>>> agent1.sinks.avroSink2-2-1.port = <port>
>>>>>>>>>> agent1.sinks.avroSink2-2-1.connect-timeout = 300000
>>>>>>>>>> agent1.sinks.avroSink2-2-1.batch-size = 4000
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> agent1.sinks.avroSink2-3-1.type = avro
>>>>>>>>>> agent1.sinks.avroSink2-3-1.channel = ch2
>>>>>>>>>> agent1.sinks.avroSink2-3-1.hostname = <hostname>
>>>>>>>>>> agent1.sinks.avroSink2-3-1.port = <port>
>>>>>>>>>> agent1.sinks.avroSink2-3-1.connect-timeout = 300000
>>>>>>>>>> agent1.sinks.avroSink2-3-1.batch-size = 4000
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> agent1.sinks.avroSink2-4-1.type = avro
>>>>>>>>>> agent1.sinks.avroSink2-4-1.channel = ch2
>>>>>>>>>> agent1.sinks.avroSink2-4-1.hostname = <hostname>
>>>>>>>>>> agent1.sinks.avroSink2-4-1.port = <port>
>>>>>>>>>> agent1.sinks.avroSink2-4-1.connect-timeout = 300000
>>>>>>>>>> agent1.sinks.avroSink2-4-1.batch-size = 4000
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>>> which round robin between different hops
>>>>>>>>>> agent1.sinkgroups.group2.sinks = avroSink2-1-1 avroSink2-2-1
>>>>>>>>>> avroSink2-3-1 avroSink2-4-1
>>>>>>>>>> agent1.sinkgroups.group2.processor.type = load_balance
>>>>>>>>>> agent1.sinkgroups.group2.processor.selector = ROUND_ROBIN
>>>>>>>>>> agent1.sinkgroups.group2.processor.backoff = true
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> #End of set
>>>>>>>>>>
>>>>>>>>>> # Define an Avro source named avroSource3
>>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> agent1.sources.avroSource3.channels = ch1
>>>>>>>>>> agent1.sources.avroSource3.type = avro
>>>>>>>>>> agent1.sources.avroSource3.bind = 0.0.0.0
>>>>>>>>>> agent1.sources.avroSource3.port = <port>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> agent1.sinks.avroSink3-1-1.type = avro
>>>>>>>>>> agent1.sinks.avroSink3-1-1.channel = ch1
>>>>>>>>>> agent1.sinks.avroSink3-1-1.hostname = <hostname>
>>>>>>>>>> agent1.sinks.avroSink3-1-1.port = <port>
>>>>>>>>>> agent1.sinks.avroSink3-1-1.connect-timeout = 300000
>>>>>>>>>> agent1.sinks.avroSink3-1-1.batch-size = 4000
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> agent1.sinks.avroSink3-2-1.type = avro
>>>>>>>>>> agent1.sinks.avroSink3-2-1.channel = ch1
>>>>>>>>>> agent1.sinks.avroSink3-2-1.hostname = <hostname>
>>>>>>>>>> agent1.sinks.avroSink3-2-1.port = <port>
>>>>>>>>>> agent1.sinks.avroSink3-2-1.connect-timeout = 300000
>>>>>>>>>> agent1.sinks.avroSink3-2-1.batch-size = 4000
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> agent1.sinks.avroSink3-3-1.type = avro
>>>>>>>>>> agent1.sinks.avroSink3-3-1.channel = ch1
>>>>>>>>>> agent1.sinks.avroSink3-3-1.hostname = <hostname>
>>>>>>>>>> agent1.sinks.avroSink3-3-1.port = <port>
>>>>>>>>>> agent1.sinks.avroSink3-3-1.connect-timeout = 300000
>>>>>>>>>> agent1.sinks.avroSink3-3-1.batch-size = 4000
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> agent1.sinks.avroSink3-4-1.type = avro
>>>>>>>>>> agent1.sinks.avroSink3-4-1.channel = ch1
>>>>>>>>>> agent1.sinks.avroSink3-4-1.hostname = <hostname>
>>>>>>>>>> agent1.sinks.avroSink3-4-1.port = <port>
>>>>>>>>>> agent1.sinks.avroSink3-4-1.connect-timeout = 300000
>>>>>>>>>> agent1.sinks.avroSink3-4-1.batch-size = 4000
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>>> which round robin between different hops
>>>>>>>>>> agent1.sinkgroups.group3.sinks = avroSink3-1-1 avroSink3-2-1
>>>>>>>>>> avroSink3-3-1 avroSink3-4-1
>>>>>>>>>> agent1.sinkgroups.group3.processor.type = load_balance
>>>>>>>>>> agent1.sinkgroups.group3.processor.selector = ROUND_ROBIN
>>>>>>>>>> agent1.sinkgroups.group3.processor.backoff = true
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> #End of set
>>>>>>>>>>
>>>>>>>>>> # Define an Avro source named avroSource4
>>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> agent1.sources.avroSource4.channels = ch2
>>>>>>>>>> agent1.sources.avroSource4.type = avro
>>>>>>>>>> agent1.sources.avroSource4.bind = 0.0.0.0
>>>>>>>>>> agent1.sources.avroSource4.port = <port>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> agent1.sinks.avroSink4-1-1.type = avro
>>>>>>>>>> agent1.sinks.avroSink4-1-1.channel = ch2
>>>>>>>>>> agent1.sinks.avroSink4-1-1.hostname = <hostname>
>>>>>>>>>> agent1.sinks.avroSink4-1-1.port = <port>
>>>>>>>>>> agent1.sinks.avroSink4-1-1.connect-timeout = 300000
>>>>>>>>>> agent1.sinks.avroSink4-1-1.batch-size = 4000
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> agent1.sinks.avroSink4-2-1.type = avro
>>>>>>>>>> agent1.sinks.avroSink4-2-1.channel = ch2
>>>>>>>>>> agent1.sinks.avroSink4-2-1.hostname = <hostname>
>>>>>>>>>> agent1.sinks.avroSink4-2-1.port = <port>
>>>>>>>>>> agent1.sinks.avroSink4-2-1.connect-timeout = 300000
>>>>>>>>>> agent1.sinks.avroSink4-2-1.batch-size = 4000
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> agent1.sinks.avroSink4-3-1.type = avro
>>>>>>>>>> agent1.sinks.avroSink4-3-1.channel = ch2
>>>>>>>>>> agent1.sinks.avroSink4-3-1.hostname = <hostname>
>>>>>>>>>> agent1.sinks.avroSink4-3-1.port = <port>
>>>>>>>>>> agent1.sinks.avroSink4-3-1.connect-timeout = 300000
>>>>>>>>>> agent1.sinks.avroSink4-3-1.batch-size = 4000
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> agent1.sinks.avroSink4-4-1.type = avro
>>>>>>>>>> agent1.sinks.avroSink4-4-1.channel = ch2
>>>>>>>>>> agent1.sinks.avroSink4-4-1.hostname = <hostname>
>>>>>>>>>> agent1.sinks.avroSink4-4-1.port = <port>
>>>>>>>>>> agent1.sinks.avroSink4-4-1.connect-timeout = 300000
>>>>>>>>>> agent1.sinks.avroSink4-4-1.batch-size = 4000
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>>> which round robin between different hops
>>>>>>>>>> agent1.sinkgroups.group4.sinks = avroSink4-1-1 avroSink4-2-1
>>>>>>>>>> avroSink4-3-1 avroSink4-4-1
>>>>>>>>>> agent1.sinkgroups.group4.processor.type = load_balance
>>>>>>>>>> agent1.sinkgroups.group4.processor.selector = ROUND_ROBIN
>>>>>>>>>> agent1.sinkgroups.group4.processor.backoff = true
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> #End of set
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> # Finally, now that we've defined all of our components, tell
>>>>>>>>>> # agent1 which ones we want to activate.
>>>>>>>>>> agent1.channels = ch1 ch2
>>>>>>>>>> agent1.sources = avroSource1 avroSource2 avroSource3 avroSource4
>>>>>>>>>> agent1.sinks = avroSink1-1-1 avroSink1-2-1 avroSink1-3-1
>>>>>>>>>> avroSink1-4-1 avroSink2-1-1 avroSink2-2-1 avroSink2-3-1 avroSink2-4-1
>>>>>>>>>> avroSink3-1-1 avroSink3-2-1 avroSink3-3-1 avroSink3-4-1 avroSink4-1-1
>>>>>>>>>> avroSink4-2-1 avroSink4-3-1 avroSink4-4-1
>>>>>>>>>> agent1.sinkgroups = group1 group2 group3 group4
>>>>>>>>>>
>>>>>>>>>> There are two disks on the box /flume1 and /flume2
>>>>>>>>>> 32 cores of CPU, 32 GB RAM, 1GbPS ethernet.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Aug 13, 2013 at 7:22 PM, Brock Noland <brock@cloudera.com
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>> Can you share your conf file?
>>>>>>>>>>> On Aug 13, 2013 9:19 PM, "Hari Shreedharan" <
>>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>  Even though the writes are done per batch, they don't go to
>>>>>>>>>>>> disk rightaway - commits are the only ones which actually cause an fsync -
>>>>>>>>>>>> which is when writes actually go to disk.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Hari
>>>>>>>>>>>>
>>>>>>>>>>>> On Tuesday, August 13, 2013 at 7:06 PM, Pankaj Gupta wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Looking at the code it seems like the lock and the i/o is done
>>>>>>>>>>>> per event and not for a batch. Is that correct? If that is the case then it
>>>>>>>>>>>> seems like there is a lot of overhead per event. The throughput I'm seeing
>>>>>>>>>>>> is 1 - 1.5 MBps per disk which is way below the sequential read/write
>>>>>>>>>>>> capacity of the disk which is easily over 50MBps. Adding more sinks doesn't
>>>>>>>>>>>> help, they just block waiting for the queue to become free. CPU usage is
>>>>>>>>>>>> 20%, there is enough RAM for page cache so that no read is going to disk.
>>>>>>>>>>>> The queue seems to be the bottleneck. What is the throughput I should
>>>>>>>>>>>> expect per disk?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland <
>>>>>>>>>>>> brock@cloudera.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> The lock is per file. Adding more directories to the channel
>>>>>>>>>>>> will cause more files to be created. Of course you'll need additional disks
>>>>>>>>>>>> behind those directories to see any performance increase.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <
>>>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>  Yes, IO is done inside locks to avoid multiple takes and puts
>>>>>>>>>>>> getting written out at the same time. Even though Java makes sure the
>>>>>>>>>>>> writes are serialized, Flume still needs to keep track of some counters
>>>>>>>>>>>> etc, so the lock is required. Note that the lock you are talking about is
>>>>>>>>>>>>  in the LogFile class, which represents a single file - so even if the
>>>>>>>>>>>> write is inside that lock (which is also inside that class itself) that
>>>>>>>>>>>>  does not cause any contention - because the lock is just preventing two IO
>>>>>>>>>>>> ops to happen at the same time.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Hari
>>>>>>>>>>>>
>>>>>>>>>>>> On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> It seems like some i/o is done inside the lock, which means
>>>>>>>>>>>> that time for taking a lock is proportional to the time for i/o and thus it
>>>>>>>>>>>> becomes a problem. I apologize in advance if I am wrong but the call stack
>>>>>>>>>>>> and behavior I'm seeing seems to suggest that. Specifically, it seems that
>>>>>>>>>>>> we do a write while inside take:
>>>>>>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>>>>>>>> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>         at sun.nio.ch.NativeThread.current(Native Method)
>>>>>>>>>>>>         at
>>>>>>>>>>>> sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>>>>>>>>>>>>         at
>>>>>>>>>>>> sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>>>>>>>>>>>>         - locked <0x00000005190ec998> (a java.lang.Object)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>>>>>>>>>>>>         - locked <0x0000000519111590> (a
>>>>>>>>>>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>>>>>>>>>>>         at org.apache.flume.channel.file.Log.take(Log.java:550)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <
>>>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Since the channel is designed to make sure that events are not
>>>>>>>>>>>> duplicated to multiple sinks, and to protect against corruption due to
>>>>>>>>>>>> concurrency issues, we do not need the locking in the channel's flume event
>>>>>>>>>>>> queue. It is unlikely that locking is what is causing performance issues
>>>>>>>>>>>> because the channel is heavily I/O bound. If you take a series of thread
>>>>>>>>>>>> dumps, you will probably see that those threads are moving forward and the
>>>>>>>>>>>> ones reading/writing from/to disk are the ones which are slower. These
>>>>>>>>>>>> locks are unlikely to hit performance much.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Hari
>>>>>>>>>>>>
>>>>>>>>>>>> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Spent some more time debugging issues with FileChannel. The
>>>>>>>>>>>> problem seems to lock contention reading from FlumeEventQueue:
>>>>>>>>>>>>
>>>>>>>>>>>> I see a lot of threads like this:
>>>>>>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>>>>>>>> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
>>>>>>>>>>>> [0x00007f821afee000]
>>>>>>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>>>>>>>>>>>>         - waiting to lock <0x0000000518ee4c90> (a
>>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I have two file channels and 8 Avro Sinks per file channel. I
>>>>>>>>>>>> added more sinks because they weren't draining fast enough. It seems like
>>>>>>>>>>>> they send the batch then wait for an ack before sending again, thus sends
>>>>>>>>>>>> are not pipelined and having more sinks seemed like a good way of getting
>>>>>>>>>>>> some parallelism.
>>>>>>>>>>>>
>>>>>>>>>>>> Here's the full stack trace:
>>>>>>>>>>>> 2013-08-13 15:30:32
>>>>>>>>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02
>>>>>>>>>>>> mixed mode):
>>>>>>>>>>>>
>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
>>>>>>>>>>>>    java.lang.Thread.State: WAITING (parking)
>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>> - parking to wait for  <0x000000056d932120> (a
>>>>>>>>>>>> java.util.concurrent.CountDownLatch$Sync)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
>>>>>>>>>>>> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
>>>>>>>>>>>>  at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
>>>>>>>>>>>> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
>>>>>>>>>>>> at $Proxy7.appendBatch(Unknown Source)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000
>>>>>>>>>>>> nid=0x41a1 waiting on condition [0x0000000000000000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>>
>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>> - parking to wait for  <0x00000005194c3340> (a
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>> - parking to wait for  <0x00000005191a16d8> (a
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>> - parking to wait for  <0x0000000519174e08> (a
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>> - parking to wait for  <0x0000000519174e98> (a
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>> - parking to wait for  <0x00000005195f9670> (a
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>> - parking to wait for  <0x0000000519174ce8> (a
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>> - parking to wait for  <0x00000005191a1918> (a
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>> - parking to wait for  <0x00000005191e7f60> (a
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153
>>>>>>>>>>>> runnable [0x00007f8210e4d000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>  at
>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>  - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
>>>>>>>>>>>> - locked <0x000000054a2917a8> (a
>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>  - locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>> - parking to wait for  <0x0000000519174f58> (a
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146
>>>>>>>>>>>> runnable [0x00007f8210f4e000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>  at
>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>  - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
>>>>>>>>>>>> - locked <0x0000000546e5dd28> (a
>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>  - locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118
>>>>>>>>>>>> runnable [0x00007f821104f000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>  at
>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>  - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
>>>>>>>>>>>> - locked <0x0000000544f763c0> (a
>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>  - locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111
>>>>>>>>>>>> runnable [0x00007f8211150000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>  at
>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>  - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>> - locked <0x000000054473d5a8> (a
>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>  - locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110
>>>>>>>>>>>> runnable [0x00007f8211251000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>  at
>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>  - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
>>>>>>>>>>>> - locked <0x00000005437f4f70> (a
>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>  - locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f
>>>>>>>>>>>> runnable [0x00007f8211352000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>  at
>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>  - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
>>>>>>>>>>>> - locked <0x00000005422a9018> (a
>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>  - locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e
>>>>>>>>>>>> runnable [0x00007f8211453000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>  at
>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>  - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
>>>>>>>>>>>> - locked <0x0000000541063d50> (a
>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>  - locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d
>>>>>>>>>>>> runnable [0x00007f8211554000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>  at
>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>  - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
>>>>>>>>>>>> - locked <0x0000000540dc1110> (a
>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>  - locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a
>>>>>>>>>>>> runnable [0x00007f8211655000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>  at
>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>  - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
>>>>>>>>>>>> - locked <0x000000053fbbc3b0> (a
>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>  - locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109
>>>>>>>>>>>> runnable [0x00007f8211756000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>  at
>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>  - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
>>>>>>>>>>>> - locked <0x000000053f812860> (a
>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>  - locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd
>>>>>>>>>>>> runnable [0x00007f8211857000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>  at
>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>  - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
>>>>>>>>>>>> - locked <0x000000053e6e5570> (a
>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>  - locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>> - parking to wait for  <0x0000000519175138> (a
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>> - parking to wait for  <0x0000000519174c28> (a
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>> - parking to wait for  <0x00000005191751c8> (a
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0
>>>>>>>>>>>> runnable [0x00007f8211c5b000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>  at
>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>  - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
>>>>>>>>>>>> - locked <0x0000000536579db0> (a
>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>  - locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>> - parking to wait for  <0x00000005191750a8> (a
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1
>>>>>>>>>>>> runnable [0x00007f8211e5d000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>  at
>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>  - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>> - locked <0x00000005313943a8> (a
>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>  - locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>> - parking to wait for  <0x000000051936f460> (a
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>> - parking to wait for  <0x000000051936dfa8> (a
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de
>>>>>>>>>>>> runnable [0x00007f8212160000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>  at
>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>  - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
>>>>>>>>>>>> - locked <0x000000052dad4468> (a
>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>  - locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd
>>>>>>>>>>>> runnable [0x00007f8212261000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>  at
>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>  - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
>>>>>>>>>>>> - locked <0x000000052c73c5f0> (a
>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>  - locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc
>>>>>>>>>>>> runnable [0x00007f8212362000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>  at
>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>  - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>> - locked <0x000000052b8da998> (a
>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>  - locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db
>>>>>>>>>>>> waiting for monitor entry [0x00007f8212462000]
>>>>>>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>>>>>>>>>>>> - waiting to lock <0x0000000518ee4c90> (a
>>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>>>>>>>>>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>>>>>>>>>>>  at
>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>>>>>>>>>>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>> - parking to wait for  <0x000000051936df18> (a
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>>> tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000]
>>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>>> - parking to wait for  <0x0000000518fe7ae8> (a
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8
>>>>>>>>>>>> runnable [0x00007f8212766000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>  at
>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>  - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>> - locked <0x00000005263d6ba8> (a
>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>  - locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7
>>>>>>>>>>>> runnable [0x00007f8212867000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>  at
>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>  - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
>>>>>>>>>>>> - locked <0x0000000525d1d0e0> (a
>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>  - locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6
>>>>>>>>>>>> runnable [0x00007f8212968000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>  at
>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>  - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>> - locked <0x0000000525a689d8> (a
>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>  - locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5
>>>>>>>>>>>> runnable [0x00007f8212a69000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>  at
>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>  - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>> - locked <0x0000000524e1e7d8> (a
>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>  - locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>  at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>> at
>>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>
>>>>>>>>>>>> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4
>>>>>>>>>>>> runnable [0x00007f8212b6a000]
>>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>>  at
>>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>>  - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
>>>>>>>>>>>> - locked <0x0000000524d437a8> (a
>>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>>  - locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>>
>>>>>>>>>>>> ...
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>>
>>>> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 |
>>>> pankaj@brightroll.com
>>>>
>>>> Pankaj Gupta | Software Engineer
>>>>
>>>> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>>>>
>>>>
>>>> United States | Canada | United Kingdom | Germany
>>>>
>>>>
>>>> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
>>>> !
>>>>
>>>
>>>
>>>
>>> --
>>> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>>>
>>
>>
>>
>> --
>>
>>
>> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 |
>> pankaj@brightroll.com
>>
>> Pankaj Gupta | Software Engineer
>>
>> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>>
>>
>> United States | Canada | United Kingdom | Germany
>>
>>
>> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
>> !
>>
>
>
>
> --
>
>
> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com
>
> Pankaj Gupta | Software Engineer
>
> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>
>
> United States | Canada | United Kingdom | Germany
>
>
> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
> !
>



-- 


*P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com

Pankaj Gupta | Software Engineer

*BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com


United States | Canada | United Kingdom | Germany


We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
!

Re: Lock contention in FileChannel

[Pankaj Gupta <pa...@brightroll.com>]

With Null Sink, in the call stack I see a lot of these:
"SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
tid=0x00007feb18001000 nid=0x356b runnable [0x00007feb9e897000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.FileChannelImpl.force0(Native Method)
        at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:348)
        at
org.apache.flume.channel.file.LogFile$Writer.sync(LogFile.java:258)
        at
org.apache.flume.channel.file.LogFile$Writer.commit(LogFile.java:225)
        - locked <0x0000000518c97650> (a
org.apache.flume.channel.file.LogFileV3$Writer)
        at org.apache.flume.channel.file.Log.commit(Log.java:758)
        at org.apache.flume.channel.file.Log.commitTake(Log.java:640)
        at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:557)
        at
org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
        at org.apache.flume.sink.NullSink.process(NullSink.java:100)
        at
org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
        at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:662)




On Wed, Aug 14, 2013 at 11:57 AM, Pankaj Gupta <pa...@brightroll.com>wrote:

> I tried increasing the dataDirs to 2 and 4 per disk but doesn't seem to
> help much. I then replaced the avro sink with null sinks and events are
> still filling up in the channel. I tried with both 2 and 4 dataDirs per
> disk and null sink, still don't get throughput higher than 1.5 MBps.
>
>
> On Tue, Aug 13, 2013 at 8:30 PM, Brock Noland <br...@cloudera.com> wrote:
>
>> Increasing the number of file channels will result in more checkpoints.
>> Therefore there will be more io than simply increasing the number of
>> dataDirs.  However, this might be a case where it'd be nice to relax the
>> file channel data consistency constraints a little to get increased
>> throughput. That feature does not exist at present.
>>
>>
>> On Tue, Aug 13, 2013 at 10:16 PM, Pankaj Gupta <pa...@brightroll.com>wrote:
>>
>>> I did try increasing number of FileChannels. At 2 FileChannels per disk
>>> performance seemed to be 25% better. At 4 FileChannels per disk performance
>>> dropped to even below 1 FileChannel per disk. I will try increasing the
>>> dataDirs tomorrow.
>>>
>>>
>>> On Tue, Aug 13, 2013 at 8:06 PM, Brock Noland <br...@cloudera.com>wrote:
>>>
>>>> dataDirs is a comma separated list. Try 3-4 directories and then the
>>>> same test.
>>>> On Aug 13, 2013 9:58 PM, "Pankaj Gupta" <pa...@brightroll.com> wrote:
>>>>
>>>>> Both disks were at around 15-25%.
>>>>>
>>>>>
>>>>> On Tue, Aug 13, 2013 at 7:54 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>
>>>>>> Gotcha. When you run tge test what is tye disk utilization
>>>>>> percentage? Iostat can be used for this.
>>>>>> On Aug 13, 2013 9:47 PM, "Pankaj Gupta" <pa...@brightroll.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Those are the boxes we want to collect data from. They run flume and
>>>>>>> send data through their avro sinks to the avro source on this box. We are
>>>>>>> getting data at a pretty good rate and the problem is in fact that the
>>>>>>> events don't drain from the FileChannel fast enough and the channel fill
>>>>>>> percentage keeps getting higher.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 13, 2013 at 7:41 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>>>
>>>>>>>> What is sending the events to the avro source?
>>>>>>>> On Aug 13, 2013 9:34 PM, "Pankaj Gupta" <pa...@brightroll.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Here's the config:
>>>>>>>>> # define channels, one for each disk
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.channels.ch1.type = FILE
>>>>>>>>> agent1.channels.ch1.checkpointDir = /flume1/checkpoint
>>>>>>>>> agent1.channels.ch1.dataDirs = /flume1/data
>>>>>>>>> agent1.channels.ch1.maxFileSize = 375809638400
>>>>>>>>> agent1.channels.ch1.capacity = 75000000
>>>>>>>>> agent1.channels.ch1.transactionCapacity = 4000
>>>>>>>>>
>>>>>>>>> agent1.channels.ch2.type = FILE
>>>>>>>>> agent1.channels.ch2.checkpointDir = /flume2/checkpoint
>>>>>>>>> agent1.channels.ch2.dataDirs = /flume2/data
>>>>>>>>> agent1.channels.ch2.maxFileSize = 375809638400
>>>>>>>>> agent1.channels.ch2.capacity = 75000000
>>>>>>>>> agent1.channels.ch2.transactionCapacity = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # Define an Avro source named avroSource1
>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sources.avroSource1.channels = ch1
>>>>>>>>> agent1.sources.avroSource1.type = avro
>>>>>>>>> agent1.sources.avroSource1.bind = 0.0.0.0
>>>>>>>>> agent1.sources.avroSource1.port = <port>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink1-1-1.type = avro
>>>>>>>>> agent1.sinks.avroSink1-1-1.channel = ch1
>>>>>>>>> agent1.sinks.avroSink1-1-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink1-1-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink1-1-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink1-1-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink1-2-1.type = avro
>>>>>>>>> agent1.sinks.avroSink1-2-1.channel = ch1
>>>>>>>>> agent1.sinks.avroSink1-2-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink1-2-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink1-2-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink1-2-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink1-3-1.type = avro
>>>>>>>>> agent1.sinks.avroSink1-3-1.channel = ch1
>>>>>>>>> agent1.sinks.avroSink1-3-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink1-3-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink1-3-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink1-3-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink1-4-1.type = avro
>>>>>>>>> agent1.sinks.avroSink1-4-1.channel = ch1
>>>>>>>>> agent1.sinks.avroSink1-4-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink1-4-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink1-4-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink1-4-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>> which round robin between different hops
>>>>>>>>> agent1.sinkgroups.group1.sinks = avroSink1-1-1 avroSink1-2-1
>>>>>>>>> avroSink1-3-1 avroSink1-4-1
>>>>>>>>> agent1.sinkgroups.group1.processor.type = load_balance
>>>>>>>>> agent1.sinkgroups.group1.processor.selector = ROUND_ROBIN
>>>>>>>>> agent1.sinkgroups.group1.processor.backoff = true
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> #End of set
>>>>>>>>>
>>>>>>>>> # Define an Avro source named avroSource2
>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sources.avroSource2.channels = ch2
>>>>>>>>> agent1.sources.avroSource2.type = avro
>>>>>>>>> agent1.sources.avroSource2.bind = 0.0.0.0
>>>>>>>>> agent1.sources.avroSource2.port = <port>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink2-1-1.type = avro
>>>>>>>>> agent1.sinks.avroSink2-1-1.channel = ch2
>>>>>>>>> agent1.sinks.avroSink2-1-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink2-1-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink2-1-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink2-1-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink2-2-1.type = avro
>>>>>>>>> agent1.sinks.avroSink2-2-1.channel = ch2
>>>>>>>>> agent1.sinks.avroSink2-2-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink2-2-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink2-2-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink2-2-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink2-3-1.type = avro
>>>>>>>>> agent1.sinks.avroSink2-3-1.channel = ch2
>>>>>>>>> agent1.sinks.avroSink2-3-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink2-3-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink2-3-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink2-3-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink2-4-1.type = avro
>>>>>>>>> agent1.sinks.avroSink2-4-1.channel = ch2
>>>>>>>>> agent1.sinks.avroSink2-4-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink2-4-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink2-4-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink2-4-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>> which round robin between different hops
>>>>>>>>> agent1.sinkgroups.group2.sinks = avroSink2-1-1 avroSink2-2-1
>>>>>>>>> avroSink2-3-1 avroSink2-4-1
>>>>>>>>> agent1.sinkgroups.group2.processor.type = load_balance
>>>>>>>>> agent1.sinkgroups.group2.processor.selector = ROUND_ROBIN
>>>>>>>>> agent1.sinkgroups.group2.processor.backoff = true
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> #End of set
>>>>>>>>>
>>>>>>>>> # Define an Avro source named avroSource3
>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sources.avroSource3.channels = ch1
>>>>>>>>> agent1.sources.avroSource3.type = avro
>>>>>>>>> agent1.sources.avroSource3.bind = 0.0.0.0
>>>>>>>>> agent1.sources.avroSource3.port = <port>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink3-1-1.type = avro
>>>>>>>>> agent1.sinks.avroSink3-1-1.channel = ch1
>>>>>>>>> agent1.sinks.avroSink3-1-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink3-1-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink3-1-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink3-1-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink3-2-1.type = avro
>>>>>>>>> agent1.sinks.avroSink3-2-1.channel = ch1
>>>>>>>>> agent1.sinks.avroSink3-2-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink3-2-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink3-2-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink3-2-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink3-3-1.type = avro
>>>>>>>>> agent1.sinks.avroSink3-3-1.channel = ch1
>>>>>>>>> agent1.sinks.avroSink3-3-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink3-3-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink3-3-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink3-3-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink3-4-1.type = avro
>>>>>>>>> agent1.sinks.avroSink3-4-1.channel = ch1
>>>>>>>>> agent1.sinks.avroSink3-4-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink3-4-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink3-4-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink3-4-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>> which round robin between different hops
>>>>>>>>> agent1.sinkgroups.group3.sinks = avroSink3-1-1 avroSink3-2-1
>>>>>>>>> avroSink3-3-1 avroSink3-4-1
>>>>>>>>> agent1.sinkgroups.group3.processor.type = load_balance
>>>>>>>>> agent1.sinkgroups.group3.processor.selector = ROUND_ROBIN
>>>>>>>>> agent1.sinkgroups.group3.processor.backoff = true
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> #End of set
>>>>>>>>>
>>>>>>>>> # Define an Avro source named avroSource4
>>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sources.avroSource4.channels = ch2
>>>>>>>>> agent1.sources.avroSource4.type = avro
>>>>>>>>> agent1.sources.avroSource4.bind = 0.0.0.0
>>>>>>>>> agent1.sources.avroSource4.port = <port>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink4-1-1.type = avro
>>>>>>>>> agent1.sinks.avroSink4-1-1.channel = ch2
>>>>>>>>> agent1.sinks.avroSink4-1-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink4-1-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink4-1-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink4-1-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink4-2-1.type = avro
>>>>>>>>> agent1.sinks.avroSink4-2-1.channel = ch2
>>>>>>>>> agent1.sinks.avroSink4-2-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink4-2-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink4-2-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink4-2-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink4-3-1.type = avro
>>>>>>>>> agent1.sinks.avroSink4-3-1.channel = ch2
>>>>>>>>> agent1.sinks.avroSink4-3-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink4-3-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink4-3-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink4-3-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> agent1.sinks.avroSink4-4-1.type = avro
>>>>>>>>> agent1.sinks.avroSink4-4-1.channel = ch2
>>>>>>>>> agent1.sinks.avroSink4-4-1.hostname = <hostname>
>>>>>>>>> agent1.sinks.avroSink4-4-1.port = <port>
>>>>>>>>> agent1.sinks.avroSink4-4-1.connect-timeout = 300000
>>>>>>>>> agent1.sinks.avroSink4-4-1.batch-size = 4000
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>>> which round robin between different hops
>>>>>>>>> agent1.sinkgroups.group4.sinks = avroSink4-1-1 avroSink4-2-1
>>>>>>>>> avroSink4-3-1 avroSink4-4-1
>>>>>>>>> agent1.sinkgroups.group4.processor.type = load_balance
>>>>>>>>> agent1.sinkgroups.group4.processor.selector = ROUND_ROBIN
>>>>>>>>> agent1.sinkgroups.group4.processor.backoff = true
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> #End of set
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> # Finally, now that we've defined all of our components, tell
>>>>>>>>> # agent1 which ones we want to activate.
>>>>>>>>> agent1.channels = ch1 ch2
>>>>>>>>> agent1.sources = avroSource1 avroSource2 avroSource3 avroSource4
>>>>>>>>> agent1.sinks = avroSink1-1-1 avroSink1-2-1 avroSink1-3-1
>>>>>>>>> avroSink1-4-1 avroSink2-1-1 avroSink2-2-1 avroSink2-3-1 avroSink2-4-1
>>>>>>>>> avroSink3-1-1 avroSink3-2-1 avroSink3-3-1 avroSink3-4-1 avroSink4-1-1
>>>>>>>>> avroSink4-2-1 avroSink4-3-1 avroSink4-4-1
>>>>>>>>> agent1.sinkgroups = group1 group2 group3 group4
>>>>>>>>>
>>>>>>>>> There are two disks on the box /flume1 and /flume2
>>>>>>>>> 32 cores of CPU, 32 GB RAM, 1GbPS ethernet.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Aug 13, 2013 at 7:22 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>>>>>
>>>>>>>>>> Can you share your conf file?
>>>>>>>>>> On Aug 13, 2013 9:19 PM, "Hari Shreedharan" <
>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>
>>>>>>>>>>>  Even though the writes are done per batch, they don't go to
>>>>>>>>>>> disk rightaway - commits are the only ones which actually cause an fsync -
>>>>>>>>>>> which is when writes actually go to disk.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Hari
>>>>>>>>>>>
>>>>>>>>>>> On Tuesday, August 13, 2013 at 7:06 PM, Pankaj Gupta wrote:
>>>>>>>>>>>
>>>>>>>>>>> Looking at the code it seems like the lock and the i/o is done
>>>>>>>>>>> per event and not for a batch. Is that correct? If that is the case then it
>>>>>>>>>>> seems like there is a lot of overhead per event. The throughput I'm seeing
>>>>>>>>>>> is 1 - 1.5 MBps per disk which is way below the sequential read/write
>>>>>>>>>>> capacity of the disk which is easily over 50MBps. Adding more sinks doesn't
>>>>>>>>>>> help, they just block waiting for the queue to become free. CPU usage is
>>>>>>>>>>> 20%, there is enough RAM for page cache so that no read is going to disk.
>>>>>>>>>>> The queue seems to be the bottleneck. What is the throughput I should
>>>>>>>>>>> expect per disk?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland <
>>>>>>>>>>> brock@cloudera.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> The lock is per file. Adding more directories to the channel
>>>>>>>>>>> will cause more files to be created. Of course you'll need additional disks
>>>>>>>>>>> behind those directories to see any performance increase.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <
>>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>  Yes, IO is done inside locks to avoid multiple takes and puts
>>>>>>>>>>> getting written out at the same time. Even though Java makes sure the
>>>>>>>>>>> writes are serialized, Flume still needs to keep track of some counters
>>>>>>>>>>> etc, so the lock is required. Note that the lock you are talking about is
>>>>>>>>>>>  in the LogFile class, which represents a single file - so even if the
>>>>>>>>>>> write is inside that lock (which is also inside that class itself) that
>>>>>>>>>>>  does not cause any contention - because the lock is just preventing two IO
>>>>>>>>>>> ops to happen at the same time.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Hari
>>>>>>>>>>>
>>>>>>>>>>> On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:
>>>>>>>>>>>
>>>>>>>>>>> It seems like some i/o is done inside the lock, which means that
>>>>>>>>>>> time for taking a lock is proportional to the time for i/o and thus it
>>>>>>>>>>> becomes a problem. I apologize in advance if I am wrong but the call stack
>>>>>>>>>>> and behavior I'm seeing seems to suggest that. Specifically, it seems that
>>>>>>>>>>> we do a write while inside take:
>>>>>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>>>>>>> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>         at sun.nio.ch.NativeThread.current(Native Method)
>>>>>>>>>>>         at
>>>>>>>>>>> sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>>>>>>>>>>>         at
>>>>>>>>>>> sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>>>>>>>>>>>         - locked <0x00000005190ec998> (a java.lang.Object)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>>>>>>>>>>>         - locked <0x0000000519111590> (a
>>>>>>>>>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>>>>>>>>>>         at org.apache.flume.channel.file.Log.take(Log.java:550)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <
>>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Since the channel is designed to make sure that events are not
>>>>>>>>>>> duplicated to multiple sinks, and to protect against corruption due to
>>>>>>>>>>> concurrency issues, we do not need the locking in the channel's flume event
>>>>>>>>>>> queue. It is unlikely that locking is what is causing performance issues
>>>>>>>>>>> because the channel is heavily I/O bound. If you take a series of thread
>>>>>>>>>>> dumps, you will probably see that those threads are moving forward and the
>>>>>>>>>>> ones reading/writing from/to disk are the ones which are slower. These
>>>>>>>>>>> locks are unlikely to hit performance much.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Hari
>>>>>>>>>>>
>>>>>>>>>>> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Spent some more time debugging issues with FileChannel. The
>>>>>>>>>>> problem seems to lock contention reading from FlumeEventQueue:
>>>>>>>>>>>
>>>>>>>>>>> I see a lot of threads like this:
>>>>>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>>>>>>> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
>>>>>>>>>>> [0x00007f821afee000]
>>>>>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>>>>>>>>>>>         - waiting to lock <0x0000000518ee4c90> (a
>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>>>>>         at
>>>>>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I have two file channels and 8 Avro Sinks per file channel. I
>>>>>>>>>>> added more sinks because they weren't draining fast enough. It seems like
>>>>>>>>>>> they send the batch then wait for an ack before sending again, thus sends
>>>>>>>>>>> are not pipelined and having more sinks seemed like a good way of getting
>>>>>>>>>>> some parallelism.
>>>>>>>>>>>
>>>>>>>>>>> Here's the full stack trace:
>>>>>>>>>>> 2013-08-13 15:30:32
>>>>>>>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02
>>>>>>>>>>> mixed mode):
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
>>>>>>>>>>>    java.lang.Thread.State: WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x000000056d932120> (a
>>>>>>>>>>> java.util.concurrent.CountDownLatch$Sync)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
>>>>>>>>>>> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
>>>>>>>>>>>  at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
>>>>>>>>>>> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
>>>>>>>>>>>  at
>>>>>>>>>>> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
>>>>>>>>>>> at $Proxy7.appendBatch(Unknown Source)
>>>>>>>>>>>  at
>>>>>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000
>>>>>>>>>>> nid=0x41a1 waiting on condition [0x0000000000000000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x00000005194c3340> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x00000005191a16d8> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x0000000519174e08> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x0000000519174e98> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x00000005195f9670> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x0000000519174ce8> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x00000005191a1918> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x00000005191e7f60> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153
>>>>>>>>>>> runnable [0x00007f8210e4d000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x000000054a2917a8> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x0000000519174f58> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146
>>>>>>>>>>> runnable [0x00007f8210f4e000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x0000000546e5dd28> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118
>>>>>>>>>>> runnable [0x00007f821104f000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x0000000544f763c0> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111
>>>>>>>>>>> runnable [0x00007f8211150000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x000000054473d5a8> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110
>>>>>>>>>>> runnable [0x00007f8211251000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x00000005437f4f70> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f
>>>>>>>>>>> runnable [0x00007f8211352000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x00000005422a9018> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e
>>>>>>>>>>> runnable [0x00007f8211453000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x0000000541063d50> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d
>>>>>>>>>>> runnable [0x00007f8211554000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x0000000540dc1110> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a
>>>>>>>>>>> runnable [0x00007f8211655000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x000000053fbbc3b0> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109
>>>>>>>>>>> runnable [0x00007f8211756000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x000000053f812860> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd
>>>>>>>>>>> runnable [0x00007f8211857000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x000000053e6e5570> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x0000000519175138> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x0000000519174c28> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x00000005191751c8> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0
>>>>>>>>>>> runnable [0x00007f8211c5b000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x0000000536579db0> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x00000005191750a8> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1
>>>>>>>>>>> runnable [0x00007f8211e5d000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x00000005313943a8> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x000000051936f460> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x000000051936dfa8> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de
>>>>>>>>>>> runnable [0x00007f8212160000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x000000052dad4468> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd
>>>>>>>>>>> runnable [0x00007f8212261000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x000000052c73c5f0> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc
>>>>>>>>>>> runnable [0x00007f8212362000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x000000052b8da998> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db
>>>>>>>>>>> waiting for monitor entry [0x00007f8212462000]
>>>>>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>>>>>>>>>>> - waiting to lock <0x0000000518ee4c90> (a
>>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>>>>>  at
>>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>>>>>>>>>>  at
>>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>>>>>>>>>>  at
>>>>>>>>>>> org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>>>>>>>>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>>>>  at
>>>>>>>>>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>>>>>>>>>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>>>>>>>>>>  at
>>>>>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>>>>>  at
>>>>>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x000000051936df18> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>>> tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000]
>>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>>> - parking to wait for  <0x0000000518fe7ae8> (a
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8
>>>>>>>>>>> runnable [0x00007f8212766000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x00000005263d6ba8> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7
>>>>>>>>>>> runnable [0x00007f8212867000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x0000000525d1d0e0> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6
>>>>>>>>>>> runnable [0x00007f8212968000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x0000000525a689d8> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5
>>>>>>>>>>> runnable [0x00007f8212a69000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x0000000524e1e7d8> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>  at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>
>>>>>>>>>>> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4
>>>>>>>>>>> runnable [0x00007f8212b6a000]
>>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>>  at
>>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>>  - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
>>>>>>>>>>> - locked <0x0000000524d437a8> (a
>>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>>  - locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>>  at
>>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>> at
>>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>>
>>>>>>>>>>> ...
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>>
>>> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 |
>>> pankaj@brightroll.com
>>>
>>> Pankaj Gupta | Software Engineer
>>>
>>> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>>>
>>>
>>> United States | Canada | United Kingdom | Germany
>>>
>>>
>>> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
>>> !
>>>
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>>
>
>
>
> --
>
>
> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com
>
> Pankaj Gupta | Software Engineer
>
> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>
>
> United States | Canada | United Kingdom | Germany
>
>
> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
> !
>



-- 


*P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com

Pankaj Gupta | Software Engineer

*BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com


United States | Canada | United Kingdom | Germany


We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
!

Re: Lock contention in FileChannel

[Pankaj Gupta <pa...@brightroll.com>]

I tried increasing the dataDirs to 2 and 4 per disk but doesn't seem to
help much. I then replaced the avro sink with null sinks and events are
still filling up in the channel. I tried with both 2 and 4 dataDirs per
disk and null sink, still don't get throughput higher than 1.5 MBps.


On Tue, Aug 13, 2013 at 8:30 PM, Brock Noland <br...@cloudera.com> wrote:

> Increasing the number of file channels will result in more checkpoints.
> Therefore there will be more io than simply increasing the number of
> dataDirs.  However, this might be a case where it'd be nice to relax the
> file channel data consistency constraints a little to get increased
> throughput. That feature does not exist at present.
>
>
> On Tue, Aug 13, 2013 at 10:16 PM, Pankaj Gupta <pa...@brightroll.com>wrote:
>
>> I did try increasing number of FileChannels. At 2 FileChannels per disk
>> performance seemed to be 25% better. At 4 FileChannels per disk performance
>> dropped to even below 1 FileChannel per disk. I will try increasing the
>> dataDirs tomorrow.
>>
>>
>> On Tue, Aug 13, 2013 at 8:06 PM, Brock Noland <br...@cloudera.com> wrote:
>>
>>> dataDirs is a comma separated list. Try 3-4 directories and then the
>>> same test.
>>> On Aug 13, 2013 9:58 PM, "Pankaj Gupta" <pa...@brightroll.com> wrote:
>>>
>>>> Both disks were at around 15-25%.
>>>>
>>>>
>>>> On Tue, Aug 13, 2013 at 7:54 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>
>>>>> Gotcha. When you run tge test what is tye disk utilization percentage?
>>>>> Iostat can be used for this.
>>>>> On Aug 13, 2013 9:47 PM, "Pankaj Gupta" <pa...@brightroll.com> wrote:
>>>>>
>>>>>> Those are the boxes we want to collect data from. They run flume and
>>>>>> send data through their avro sinks to the avro source on this box. We are
>>>>>> getting data at a pretty good rate and the problem is in fact that the
>>>>>> events don't drain from the FileChannel fast enough and the channel fill
>>>>>> percentage keeps getting higher.
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 13, 2013 at 7:41 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>>
>>>>>>> What is sending the events to the avro source?
>>>>>>> On Aug 13, 2013 9:34 PM, "Pankaj Gupta" <pa...@brightroll.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Here's the config:
>>>>>>>> # define channels, one for each disk
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.channels.ch1.type = FILE
>>>>>>>> agent1.channels.ch1.checkpointDir = /flume1/checkpoint
>>>>>>>> agent1.channels.ch1.dataDirs = /flume1/data
>>>>>>>> agent1.channels.ch1.maxFileSize = 375809638400
>>>>>>>> agent1.channels.ch1.capacity = 75000000
>>>>>>>> agent1.channels.ch1.transactionCapacity = 4000
>>>>>>>>
>>>>>>>> agent1.channels.ch2.type = FILE
>>>>>>>> agent1.channels.ch2.checkpointDir = /flume2/checkpoint
>>>>>>>> agent1.channels.ch2.dataDirs = /flume2/data
>>>>>>>> agent1.channels.ch2.maxFileSize = 375809638400
>>>>>>>> agent1.channels.ch2.capacity = 75000000
>>>>>>>> agent1.channels.ch2.transactionCapacity = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> # Define an Avro source named avroSource1
>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sources.avroSource1.channels = ch1
>>>>>>>> agent1.sources.avroSource1.type = avro
>>>>>>>> agent1.sources.avroSource1.bind = 0.0.0.0
>>>>>>>> agent1.sources.avroSource1.port = <port>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink1-1-1.type = avro
>>>>>>>> agent1.sinks.avroSink1-1-1.channel = ch1
>>>>>>>> agent1.sinks.avroSink1-1-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink1-1-1.port = <port>
>>>>>>>> agent1.sinks.avroSink1-1-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink1-1-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink1-2-1.type = avro
>>>>>>>> agent1.sinks.avroSink1-2-1.channel = ch1
>>>>>>>> agent1.sinks.avroSink1-2-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink1-2-1.port = <port>
>>>>>>>> agent1.sinks.avroSink1-2-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink1-2-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink1-3-1.type = avro
>>>>>>>> agent1.sinks.avroSink1-3-1.channel = ch1
>>>>>>>> agent1.sinks.avroSink1-3-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink1-3-1.port = <port>
>>>>>>>> agent1.sinks.avroSink1-3-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink1-3-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink1-4-1.type = avro
>>>>>>>> agent1.sinks.avroSink1-4-1.channel = ch1
>>>>>>>> agent1.sinks.avroSink1-4-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink1-4-1.port = <port>
>>>>>>>> agent1.sinks.avroSink1-4-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink1-4-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>> which round robin between different hops
>>>>>>>> agent1.sinkgroups.group1.sinks = avroSink1-1-1 avroSink1-2-1
>>>>>>>> avroSink1-3-1 avroSink1-4-1
>>>>>>>> agent1.sinkgroups.group1.processor.type = load_balance
>>>>>>>> agent1.sinkgroups.group1.processor.selector = ROUND_ROBIN
>>>>>>>> agent1.sinkgroups.group1.processor.backoff = true
>>>>>>>>
>>>>>>>>
>>>>>>>> #End of set
>>>>>>>>
>>>>>>>> # Define an Avro source named avroSource2
>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sources.avroSource2.channels = ch2
>>>>>>>> agent1.sources.avroSource2.type = avro
>>>>>>>> agent1.sources.avroSource2.bind = 0.0.0.0
>>>>>>>> agent1.sources.avroSource2.port = <port>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink2-1-1.type = avro
>>>>>>>> agent1.sinks.avroSink2-1-1.channel = ch2
>>>>>>>> agent1.sinks.avroSink2-1-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink2-1-1.port = <port>
>>>>>>>> agent1.sinks.avroSink2-1-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink2-1-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink2-2-1.type = avro
>>>>>>>> agent1.sinks.avroSink2-2-1.channel = ch2
>>>>>>>> agent1.sinks.avroSink2-2-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink2-2-1.port = <port>
>>>>>>>> agent1.sinks.avroSink2-2-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink2-2-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink2-3-1.type = avro
>>>>>>>> agent1.sinks.avroSink2-3-1.channel = ch2
>>>>>>>> agent1.sinks.avroSink2-3-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink2-3-1.port = <port>
>>>>>>>> agent1.sinks.avroSink2-3-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink2-3-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink2-4-1.type = avro
>>>>>>>> agent1.sinks.avroSink2-4-1.channel = ch2
>>>>>>>> agent1.sinks.avroSink2-4-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink2-4-1.port = <port>
>>>>>>>> agent1.sinks.avroSink2-4-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink2-4-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>> which round robin between different hops
>>>>>>>> agent1.sinkgroups.group2.sinks = avroSink2-1-1 avroSink2-2-1
>>>>>>>> avroSink2-3-1 avroSink2-4-1
>>>>>>>> agent1.sinkgroups.group2.processor.type = load_balance
>>>>>>>> agent1.sinkgroups.group2.processor.selector = ROUND_ROBIN
>>>>>>>> agent1.sinkgroups.group2.processor.backoff = true
>>>>>>>>
>>>>>>>>
>>>>>>>> #End of set
>>>>>>>>
>>>>>>>> # Define an Avro source named avroSource3
>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sources.avroSource3.channels = ch1
>>>>>>>> agent1.sources.avroSource3.type = avro
>>>>>>>> agent1.sources.avroSource3.bind = 0.0.0.0
>>>>>>>> agent1.sources.avroSource3.port = <port>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink3-1-1.type = avro
>>>>>>>> agent1.sinks.avroSink3-1-1.channel = ch1
>>>>>>>> agent1.sinks.avroSink3-1-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink3-1-1.port = <port>
>>>>>>>> agent1.sinks.avroSink3-1-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink3-1-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink3-2-1.type = avro
>>>>>>>> agent1.sinks.avroSink3-2-1.channel = ch1
>>>>>>>> agent1.sinks.avroSink3-2-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink3-2-1.port = <port>
>>>>>>>> agent1.sinks.avroSink3-2-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink3-2-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink3-3-1.type = avro
>>>>>>>> agent1.sinks.avroSink3-3-1.channel = ch1
>>>>>>>> agent1.sinks.avroSink3-3-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink3-3-1.port = <port>
>>>>>>>> agent1.sinks.avroSink3-3-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink3-3-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink3-4-1.type = avro
>>>>>>>> agent1.sinks.avroSink3-4-1.channel = ch1
>>>>>>>> agent1.sinks.avroSink3-4-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink3-4-1.port = <port>
>>>>>>>> agent1.sinks.avroSink3-4-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink3-4-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>> which round robin between different hops
>>>>>>>> agent1.sinkgroups.group3.sinks = avroSink3-1-1 avroSink3-2-1
>>>>>>>> avroSink3-3-1 avroSink3-4-1
>>>>>>>> agent1.sinkgroups.group3.processor.type = load_balance
>>>>>>>> agent1.sinkgroups.group3.processor.selector = ROUND_ROBIN
>>>>>>>> agent1.sinkgroups.group3.processor.backoff = true
>>>>>>>>
>>>>>>>>
>>>>>>>> #End of set
>>>>>>>>
>>>>>>>> # Define an Avro source named avroSource4
>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sources.avroSource4.channels = ch2
>>>>>>>> agent1.sources.avroSource4.type = avro
>>>>>>>> agent1.sources.avroSource4.bind = 0.0.0.0
>>>>>>>> agent1.sources.avroSource4.port = <port>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink4-1-1.type = avro
>>>>>>>> agent1.sinks.avroSink4-1-1.channel = ch2
>>>>>>>> agent1.sinks.avroSink4-1-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink4-1-1.port = <port>
>>>>>>>> agent1.sinks.avroSink4-1-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink4-1-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink4-2-1.type = avro
>>>>>>>> agent1.sinks.avroSink4-2-1.channel = ch2
>>>>>>>> agent1.sinks.avroSink4-2-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink4-2-1.port = <port>
>>>>>>>> agent1.sinks.avroSink4-2-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink4-2-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink4-3-1.type = avro
>>>>>>>> agent1.sinks.avroSink4-3-1.channel = ch2
>>>>>>>> agent1.sinks.avroSink4-3-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink4-3-1.port = <port>
>>>>>>>> agent1.sinks.avroSink4-3-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink4-3-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink4-4-1.type = avro
>>>>>>>> agent1.sinks.avroSink4-4-1.channel = ch2
>>>>>>>> agent1.sinks.avroSink4-4-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink4-4-1.port = <port>
>>>>>>>> agent1.sinks.avroSink4-4-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink4-4-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>> which round robin between different hops
>>>>>>>> agent1.sinkgroups.group4.sinks = avroSink4-1-1 avroSink4-2-1
>>>>>>>> avroSink4-3-1 avroSink4-4-1
>>>>>>>> agent1.sinkgroups.group4.processor.type = load_balance
>>>>>>>> agent1.sinkgroups.group4.processor.selector = ROUND_ROBIN
>>>>>>>> agent1.sinkgroups.group4.processor.backoff = true
>>>>>>>>
>>>>>>>>
>>>>>>>> #End of set
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> # Finally, now that we've defined all of our components, tell
>>>>>>>> # agent1 which ones we want to activate.
>>>>>>>> agent1.channels = ch1 ch2
>>>>>>>> agent1.sources = avroSource1 avroSource2 avroSource3 avroSource4
>>>>>>>> agent1.sinks = avroSink1-1-1 avroSink1-2-1 avroSink1-3-1
>>>>>>>> avroSink1-4-1 avroSink2-1-1 avroSink2-2-1 avroSink2-3-1 avroSink2-4-1
>>>>>>>> avroSink3-1-1 avroSink3-2-1 avroSink3-3-1 avroSink3-4-1 avroSink4-1-1
>>>>>>>> avroSink4-2-1 avroSink4-3-1 avroSink4-4-1
>>>>>>>> agent1.sinkgroups = group1 group2 group3 group4
>>>>>>>>
>>>>>>>> There are two disks on the box /flume1 and /flume2
>>>>>>>> 32 cores of CPU, 32 GB RAM, 1GbPS ethernet.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Aug 13, 2013 at 7:22 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>>>>
>>>>>>>>> Can you share your conf file?
>>>>>>>>> On Aug 13, 2013 9:19 PM, "Hari Shreedharan" <
>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>
>>>>>>>>>>  Even though the writes are done per batch, they don't go to disk
>>>>>>>>>> rightaway - commits are the only ones which actually cause an fsync - which
>>>>>>>>>> is when writes actually go to disk.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Hari
>>>>>>>>>>
>>>>>>>>>> On Tuesday, August 13, 2013 at 7:06 PM, Pankaj Gupta wrote:
>>>>>>>>>>
>>>>>>>>>> Looking at the code it seems like the lock and the i/o is done
>>>>>>>>>> per event and not for a batch. Is that correct? If that is the case then it
>>>>>>>>>> seems like there is a lot of overhead per event. The throughput I'm seeing
>>>>>>>>>> is 1 - 1.5 MBps per disk which is way below the sequential read/write
>>>>>>>>>> capacity of the disk which is easily over 50MBps. Adding more sinks doesn't
>>>>>>>>>> help, they just block waiting for the queue to become free. CPU usage is
>>>>>>>>>> 20%, there is enough RAM for page cache so that no read is going to disk.
>>>>>>>>>> The queue seems to be the bottleneck. What is the throughput I should
>>>>>>>>>> expect per disk?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland <brock@cloudera.com
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>> The lock is per file. Adding more directories to the channel will
>>>>>>>>>> cause more files to be created. Of course you'll need additional disks
>>>>>>>>>> behind those directories to see any performance increase.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <
>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>
>>>>>>>>>>  Yes, IO is done inside locks to avoid multiple takes and puts
>>>>>>>>>> getting written out at the same time. Even though Java makes sure the
>>>>>>>>>> writes are serialized, Flume still needs to keep track of some counters
>>>>>>>>>> etc, so the lock is required. Note that the lock you are talking about is
>>>>>>>>>>  in the LogFile class, which represents a single file - so even if the
>>>>>>>>>> write is inside that lock (which is also inside that class itself) that
>>>>>>>>>>  does not cause any contention - because the lock is just preventing two IO
>>>>>>>>>> ops to happen at the same time.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Hari
>>>>>>>>>>
>>>>>>>>>> On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:
>>>>>>>>>>
>>>>>>>>>> It seems like some i/o is done inside the lock, which means that
>>>>>>>>>> time for taking a lock is proportional to the time for i/o and thus it
>>>>>>>>>> becomes a problem. I apologize in advance if I am wrong but the call stack
>>>>>>>>>> and behavior I'm seeing seems to suggest that. Specifically, it seems that
>>>>>>>>>> we do a write while inside take:
>>>>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>>>>>> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>         at sun.nio.ch.NativeThread.current(Native Method)
>>>>>>>>>>         at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>>>>>>>>>>         at
>>>>>>>>>> sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>>>>>>>>>>         - locked <0x00000005190ec998> (a java.lang.Object)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>>>>>>>>>>         - locked <0x0000000519111590> (a
>>>>>>>>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>>>>>>>>>         at org.apache.flume.channel.file.Log.take(Log.java:550)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <
>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Since the channel is designed to make sure that events are not
>>>>>>>>>> duplicated to multiple sinks, and to protect against corruption due to
>>>>>>>>>> concurrency issues, we do not need the locking in the channel's flume event
>>>>>>>>>> queue. It is unlikely that locking is what is causing performance issues
>>>>>>>>>> because the channel is heavily I/O bound. If you take a series of thread
>>>>>>>>>> dumps, you will probably see that those threads are moving forward and the
>>>>>>>>>> ones reading/writing from/to disk are the ones which are slower. These
>>>>>>>>>> locks are unlikely to hit performance much.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Hari
>>>>>>>>>>
>>>>>>>>>> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Spent some more time debugging issues with FileChannel. The
>>>>>>>>>> problem seems to lock contention reading from FlumeEventQueue:
>>>>>>>>>>
>>>>>>>>>> I see a lot of threads like this:
>>>>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>>>>>> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
>>>>>>>>>> [0x00007f821afee000]
>>>>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>>>>>>>>>>         - waiting to lock <0x0000000518ee4c90> (a
>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I have two file channels and 8 Avro Sinks per file channel. I
>>>>>>>>>> added more sinks because they weren't draining fast enough. It seems like
>>>>>>>>>> they send the batch then wait for an ack before sending again, thus sends
>>>>>>>>>> are not pipelined and having more sinks seemed like a good way of getting
>>>>>>>>>> some parallelism.
>>>>>>>>>>
>>>>>>>>>> Here's the full stack trace:
>>>>>>>>>> 2013-08-13 15:30:32
>>>>>>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02
>>>>>>>>>> mixed mode):
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
>>>>>>>>>>    java.lang.Thread.State: WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x000000056d932120> (a
>>>>>>>>>> java.util.concurrent.CountDownLatch$Sync)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
>>>>>>>>>> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
>>>>>>>>>>  at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
>>>>>>>>>> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
>>>>>>>>>>  at
>>>>>>>>>> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
>>>>>>>>>> at $Proxy7.appendBatch(Unknown Source)
>>>>>>>>>>  at
>>>>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
>>>>>>>>>> at
>>>>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000
>>>>>>>>>> nid=0x41a1 waiting on condition [0x0000000000000000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x00000005194c3340> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x00000005191a16d8> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x0000000519174e08> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x0000000519174e98> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x00000005195f9670> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x0000000519174ce8> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x00000005191a1918> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x00000005191e7f60> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153
>>>>>>>>>> runnable [0x00007f8210e4d000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x000000054a2917a8> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x0000000519174f58> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146
>>>>>>>>>> runnable [0x00007f8210f4e000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x0000000546e5dd28> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118
>>>>>>>>>> runnable [0x00007f821104f000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x0000000544f763c0> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111
>>>>>>>>>> runnable [0x00007f8211150000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x000000054473d5a8> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110
>>>>>>>>>> runnable [0x00007f8211251000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x00000005437f4f70> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f
>>>>>>>>>> runnable [0x00007f8211352000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x00000005422a9018> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e
>>>>>>>>>> runnable [0x00007f8211453000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x0000000541063d50> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d
>>>>>>>>>> runnable [0x00007f8211554000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x0000000540dc1110> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a
>>>>>>>>>> runnable [0x00007f8211655000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x000000053fbbc3b0> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109
>>>>>>>>>> runnable [0x00007f8211756000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x000000053f812860> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd
>>>>>>>>>> runnable [0x00007f8211857000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x000000053e6e5570> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x0000000519175138> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x0000000519174c28> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x00000005191751c8> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0
>>>>>>>>>> runnable [0x00007f8211c5b000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x0000000536579db0> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x00000005191750a8> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1
>>>>>>>>>> runnable [0x00007f8211e5d000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x00000005313943a8> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x000000051936f460> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x000000051936dfa8> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de
>>>>>>>>>> runnable [0x00007f8212160000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x000000052dad4468> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd
>>>>>>>>>> runnable [0x00007f8212261000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x000000052c73c5f0> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc
>>>>>>>>>> runnable [0x00007f8212362000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x000000052b8da998> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db
>>>>>>>>>> waiting for monitor entry [0x00007f8212462000]
>>>>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>>>> at
>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>>>>>>>>>> - waiting to lock <0x0000000518ee4c90> (a
>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>>>>  at
>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>>>>>>>>>> at
>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>>>>>>>>>  at
>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>>>>>>>>> at
>>>>>>>>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>>>>>>>>>  at
>>>>>>>>>> org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>>>>>>>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>>>>>>>>>  at
>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>>>  at
>>>>>>>>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>>>>>>>>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>>>>>>>>>  at
>>>>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>>>>  at
>>>>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x000000051936df18> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x0000000518fe7ae8> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8
>>>>>>>>>> runnable [0x00007f8212766000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x00000005263d6ba8> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7
>>>>>>>>>> runnable [0x00007f8212867000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x0000000525d1d0e0> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6
>>>>>>>>>> runnable [0x00007f8212968000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x0000000525a689d8> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5
>>>>>>>>>> runnable [0x00007f8212a69000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x0000000524e1e7d8> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4
>>>>>>>>>> runnable [0x00007f8212b6a000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x0000000524d437a8> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>
>>>>>>>>>> ...
>>>
>>>
>>
>>
>> --
>>
>>
>> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 |
>> pankaj@brightroll.com
>>
>> Pankaj Gupta | Software Engineer
>>
>> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>>
>>
>> United States | Canada | United Kingdom | Germany
>>
>>
>> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
>> !
>>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>



-- 


*P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com

Pankaj Gupta | Software Engineer

*BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com


United States | Canada | United Kingdom | Germany


We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
!

Re: Lock contention in FileChannel

[Brock Noland <br...@cloudera.com>]

Increasing the number of file channels will result in more checkpoints.
Therefore there will be more io than simply increasing the number of
dataDirs.  However, this might be a case where it'd be nice to relax the
file channel data consistency constraints a little to get increased
throughput. That feature does not exist at present.


On Tue, Aug 13, 2013 at 10:16 PM, Pankaj Gupta <pa...@brightroll.com>wrote:

> I did try increasing number of FileChannels. At 2 FileChannels per disk
> performance seemed to be 25% better. At 4 FileChannels per disk performance
> dropped to even below 1 FileChannel per disk. I will try increasing the
> dataDirs tomorrow.
>
>
> On Tue, Aug 13, 2013 at 8:06 PM, Brock Noland <br...@cloudera.com> wrote:
>
>> dataDirs is a comma separated list. Try 3-4 directories and then the same
>> test.
>> On Aug 13, 2013 9:58 PM, "Pankaj Gupta" <pa...@brightroll.com> wrote:
>>
>>> Both disks were at around 15-25%.
>>>
>>>
>>> On Tue, Aug 13, 2013 at 7:54 PM, Brock Noland <br...@cloudera.com>wrote:
>>>
>>>> Gotcha. When you run tge test what is tye disk utilization percentage?
>>>> Iostat can be used for this.
>>>> On Aug 13, 2013 9:47 PM, "Pankaj Gupta" <pa...@brightroll.com> wrote:
>>>>
>>>>> Those are the boxes we want to collect data from. They run flume and
>>>>> send data through their avro sinks to the avro source on this box. We are
>>>>> getting data at a pretty good rate and the problem is in fact that the
>>>>> events don't drain from the FileChannel fast enough and the channel fill
>>>>> percentage keeps getting higher.
>>>>>
>>>>>
>>>>> On Tue, Aug 13, 2013 at 7:41 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>
>>>>>> What is sending the events to the avro source?
>>>>>> On Aug 13, 2013 9:34 PM, "Pankaj Gupta" <pa...@brightroll.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Here's the config:
>>>>>>> # define channels, one for each disk
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> agent1.channels.ch1.type = FILE
>>>>>>> agent1.channels.ch1.checkpointDir = /flume1/checkpoint
>>>>>>> agent1.channels.ch1.dataDirs = /flume1/data
>>>>>>> agent1.channels.ch1.maxFileSize = 375809638400
>>>>>>> agent1.channels.ch1.capacity = 75000000
>>>>>>> agent1.channels.ch1.transactionCapacity = 4000
>>>>>>>
>>>>>>> agent1.channels.ch2.type = FILE
>>>>>>> agent1.channels.ch2.checkpointDir = /flume2/checkpoint
>>>>>>> agent1.channels.ch2.dataDirs = /flume2/data
>>>>>>> agent1.channels.ch2.maxFileSize = 375809638400
>>>>>>> agent1.channels.ch2.capacity = 75000000
>>>>>>> agent1.channels.ch2.transactionCapacity = 4000
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> # Define an Avro source named avroSource1
>>>>>>> # Each sink can connect to only one channel.
>>>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>>>
>>>>>>>
>>>>>>> agent1.sources.avroSource1.channels = ch1
>>>>>>> agent1.sources.avroSource1.type = avro
>>>>>>> agent1.sources.avroSource1.bind = 0.0.0.0
>>>>>>> agent1.sources.avroSource1.port = <port>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> agent1.sinks.avroSink1-1-1.type = avro
>>>>>>> agent1.sinks.avroSink1-1-1.channel = ch1
>>>>>>> agent1.sinks.avroSink1-1-1.hostname = <hostname>
>>>>>>> agent1.sinks.avroSink1-1-1.port = <port>
>>>>>>> agent1.sinks.avroSink1-1-1.connect-timeout = 300000
>>>>>>> agent1.sinks.avroSink1-1-1.batch-size = 4000
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> agent1.sinks.avroSink1-2-1.type = avro
>>>>>>> agent1.sinks.avroSink1-2-1.channel = ch1
>>>>>>> agent1.sinks.avroSink1-2-1.hostname = <hostname>
>>>>>>> agent1.sinks.avroSink1-2-1.port = <port>
>>>>>>> agent1.sinks.avroSink1-2-1.connect-timeout = 300000
>>>>>>> agent1.sinks.avroSink1-2-1.batch-size = 4000
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> agent1.sinks.avroSink1-3-1.type = avro
>>>>>>> agent1.sinks.avroSink1-3-1.channel = ch1
>>>>>>> agent1.sinks.avroSink1-3-1.hostname = <hostname>
>>>>>>> agent1.sinks.avroSink1-3-1.port = <port>
>>>>>>> agent1.sinks.avroSink1-3-1.connect-timeout = 300000
>>>>>>> agent1.sinks.avroSink1-3-1.batch-size = 4000
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> agent1.sinks.avroSink1-4-1.type = avro
>>>>>>> agent1.sinks.avroSink1-4-1.channel = ch1
>>>>>>> agent1.sinks.avroSink1-4-1.hostname = <hostname>
>>>>>>> agent1.sinks.avroSink1-4-1.port = <port>
>>>>>>> agent1.sinks.avroSink1-4-1.connect-timeout = 300000
>>>>>>> agent1.sinks.avroSink1-4-1.batch-size = 4000
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> #Add the sink groups; load-balance between each group of sinks which
>>>>>>> round robin between different hops
>>>>>>> agent1.sinkgroups.group1.sinks = avroSink1-1-1 avroSink1-2-1
>>>>>>> avroSink1-3-1 avroSink1-4-1
>>>>>>> agent1.sinkgroups.group1.processor.type = load_balance
>>>>>>> agent1.sinkgroups.group1.processor.selector = ROUND_ROBIN
>>>>>>> agent1.sinkgroups.group1.processor.backoff = true
>>>>>>>
>>>>>>>
>>>>>>> #End of set
>>>>>>>
>>>>>>> # Define an Avro source named avroSource2
>>>>>>> # Each sink can connect to only one channel.
>>>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>>>
>>>>>>>
>>>>>>> agent1.sources.avroSource2.channels = ch2
>>>>>>> agent1.sources.avroSource2.type = avro
>>>>>>> agent1.sources.avroSource2.bind = 0.0.0.0
>>>>>>> agent1.sources.avroSource2.port = <port>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> agent1.sinks.avroSink2-1-1.type = avro
>>>>>>> agent1.sinks.avroSink2-1-1.channel = ch2
>>>>>>> agent1.sinks.avroSink2-1-1.hostname = <hostname>
>>>>>>> agent1.sinks.avroSink2-1-1.port = <port>
>>>>>>> agent1.sinks.avroSink2-1-1.connect-timeout = 300000
>>>>>>> agent1.sinks.avroSink2-1-1.batch-size = 4000
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> agent1.sinks.avroSink2-2-1.type = avro
>>>>>>> agent1.sinks.avroSink2-2-1.channel = ch2
>>>>>>> agent1.sinks.avroSink2-2-1.hostname = <hostname>
>>>>>>> agent1.sinks.avroSink2-2-1.port = <port>
>>>>>>> agent1.sinks.avroSink2-2-1.connect-timeout = 300000
>>>>>>> agent1.sinks.avroSink2-2-1.batch-size = 4000
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> agent1.sinks.avroSink2-3-1.type = avro
>>>>>>> agent1.sinks.avroSink2-3-1.channel = ch2
>>>>>>> agent1.sinks.avroSink2-3-1.hostname = <hostname>
>>>>>>> agent1.sinks.avroSink2-3-1.port = <port>
>>>>>>> agent1.sinks.avroSink2-3-1.connect-timeout = 300000
>>>>>>> agent1.sinks.avroSink2-3-1.batch-size = 4000
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> agent1.sinks.avroSink2-4-1.type = avro
>>>>>>> agent1.sinks.avroSink2-4-1.channel = ch2
>>>>>>> agent1.sinks.avroSink2-4-1.hostname = <hostname>
>>>>>>> agent1.sinks.avroSink2-4-1.port = <port>
>>>>>>> agent1.sinks.avroSink2-4-1.connect-timeout = 300000
>>>>>>> agent1.sinks.avroSink2-4-1.batch-size = 4000
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> #Add the sink groups; load-balance between each group of sinks which
>>>>>>> round robin between different hops
>>>>>>> agent1.sinkgroups.group2.sinks = avroSink2-1-1 avroSink2-2-1
>>>>>>> avroSink2-3-1 avroSink2-4-1
>>>>>>> agent1.sinkgroups.group2.processor.type = load_balance
>>>>>>> agent1.sinkgroups.group2.processor.selector = ROUND_ROBIN
>>>>>>> agent1.sinkgroups.group2.processor.backoff = true
>>>>>>>
>>>>>>>
>>>>>>> #End of set
>>>>>>>
>>>>>>> # Define an Avro source named avroSource3
>>>>>>> # Each sink can connect to only one channel.
>>>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>>>
>>>>>>>
>>>>>>> agent1.sources.avroSource3.channels = ch1
>>>>>>> agent1.sources.avroSource3.type = avro
>>>>>>> agent1.sources.avroSource3.bind = 0.0.0.0
>>>>>>> agent1.sources.avroSource3.port = <port>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> agent1.sinks.avroSink3-1-1.type = avro
>>>>>>> agent1.sinks.avroSink3-1-1.channel = ch1
>>>>>>> agent1.sinks.avroSink3-1-1.hostname = <hostname>
>>>>>>> agent1.sinks.avroSink3-1-1.port = <port>
>>>>>>> agent1.sinks.avroSink3-1-1.connect-timeout = 300000
>>>>>>> agent1.sinks.avroSink3-1-1.batch-size = 4000
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> agent1.sinks.avroSink3-2-1.type = avro
>>>>>>> agent1.sinks.avroSink3-2-1.channel = ch1
>>>>>>> agent1.sinks.avroSink3-2-1.hostname = <hostname>
>>>>>>> agent1.sinks.avroSink3-2-1.port = <port>
>>>>>>> agent1.sinks.avroSink3-2-1.connect-timeout = 300000
>>>>>>> agent1.sinks.avroSink3-2-1.batch-size = 4000
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> agent1.sinks.avroSink3-3-1.type = avro
>>>>>>> agent1.sinks.avroSink3-3-1.channel = ch1
>>>>>>> agent1.sinks.avroSink3-3-1.hostname = <hostname>
>>>>>>> agent1.sinks.avroSink3-3-1.port = <port>
>>>>>>> agent1.sinks.avroSink3-3-1.connect-timeout = 300000
>>>>>>> agent1.sinks.avroSink3-3-1.batch-size = 4000
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> agent1.sinks.avroSink3-4-1.type = avro
>>>>>>> agent1.sinks.avroSink3-4-1.channel = ch1
>>>>>>> agent1.sinks.avroSink3-4-1.hostname = <hostname>
>>>>>>> agent1.sinks.avroSink3-4-1.port = <port>
>>>>>>> agent1.sinks.avroSink3-4-1.connect-timeout = 300000
>>>>>>> agent1.sinks.avroSink3-4-1.batch-size = 4000
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> #Add the sink groups; load-balance between each group of sinks which
>>>>>>> round robin between different hops
>>>>>>> agent1.sinkgroups.group3.sinks = avroSink3-1-1 avroSink3-2-1
>>>>>>> avroSink3-3-1 avroSink3-4-1
>>>>>>> agent1.sinkgroups.group3.processor.type = load_balance
>>>>>>> agent1.sinkgroups.group3.processor.selector = ROUND_ROBIN
>>>>>>> agent1.sinkgroups.group3.processor.backoff = true
>>>>>>>
>>>>>>>
>>>>>>> #End of set
>>>>>>>
>>>>>>> # Define an Avro source named avroSource4
>>>>>>> # Each sink can connect to only one channel.
>>>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>>>
>>>>>>>
>>>>>>> agent1.sources.avroSource4.channels = ch2
>>>>>>> agent1.sources.avroSource4.type = avro
>>>>>>> agent1.sources.avroSource4.bind = 0.0.0.0
>>>>>>> agent1.sources.avroSource4.port = <port>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> agent1.sinks.avroSink4-1-1.type = avro
>>>>>>> agent1.sinks.avroSink4-1-1.channel = ch2
>>>>>>> agent1.sinks.avroSink4-1-1.hostname = <hostname>
>>>>>>> agent1.sinks.avroSink4-1-1.port = <port>
>>>>>>> agent1.sinks.avroSink4-1-1.connect-timeout = 300000
>>>>>>> agent1.sinks.avroSink4-1-1.batch-size = 4000
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> agent1.sinks.avroSink4-2-1.type = avro
>>>>>>> agent1.sinks.avroSink4-2-1.channel = ch2
>>>>>>> agent1.sinks.avroSink4-2-1.hostname = <hostname>
>>>>>>> agent1.sinks.avroSink4-2-1.port = <port>
>>>>>>> agent1.sinks.avroSink4-2-1.connect-timeout = 300000
>>>>>>> agent1.sinks.avroSink4-2-1.batch-size = 4000
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> agent1.sinks.avroSink4-3-1.type = avro
>>>>>>> agent1.sinks.avroSink4-3-1.channel = ch2
>>>>>>> agent1.sinks.avroSink4-3-1.hostname = <hostname>
>>>>>>> agent1.sinks.avroSink4-3-1.port = <port>
>>>>>>> agent1.sinks.avroSink4-3-1.connect-timeout = 300000
>>>>>>> agent1.sinks.avroSink4-3-1.batch-size = 4000
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> agent1.sinks.avroSink4-4-1.type = avro
>>>>>>> agent1.sinks.avroSink4-4-1.channel = ch2
>>>>>>> agent1.sinks.avroSink4-4-1.hostname = <hostname>
>>>>>>> agent1.sinks.avroSink4-4-1.port = <port>
>>>>>>> agent1.sinks.avroSink4-4-1.connect-timeout = 300000
>>>>>>> agent1.sinks.avroSink4-4-1.batch-size = 4000
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> #Add the sink groups; load-balance between each group of sinks which
>>>>>>> round robin between different hops
>>>>>>> agent1.sinkgroups.group4.sinks = avroSink4-1-1 avroSink4-2-1
>>>>>>> avroSink4-3-1 avroSink4-4-1
>>>>>>> agent1.sinkgroups.group4.processor.type = load_balance
>>>>>>> agent1.sinkgroups.group4.processor.selector = ROUND_ROBIN
>>>>>>> agent1.sinkgroups.group4.processor.backoff = true
>>>>>>>
>>>>>>>
>>>>>>> #End of set
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> # Finally, now that we've defined all of our components, tell
>>>>>>> # agent1 which ones we want to activate.
>>>>>>> agent1.channels = ch1 ch2
>>>>>>> agent1.sources = avroSource1 avroSource2 avroSource3 avroSource4
>>>>>>> agent1.sinks = avroSink1-1-1 avroSink1-2-1 avroSink1-3-1
>>>>>>> avroSink1-4-1 avroSink2-1-1 avroSink2-2-1 avroSink2-3-1 avroSink2-4-1
>>>>>>> avroSink3-1-1 avroSink3-2-1 avroSink3-3-1 avroSink3-4-1 avroSink4-1-1
>>>>>>> avroSink4-2-1 avroSink4-3-1 avroSink4-4-1
>>>>>>> agent1.sinkgroups = group1 group2 group3 group4
>>>>>>>
>>>>>>> There are two disks on the box /flume1 and /flume2
>>>>>>> 32 cores of CPU, 32 GB RAM, 1GbPS ethernet.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 13, 2013 at 7:22 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>>>
>>>>>>>> Can you share your conf file?
>>>>>>>> On Aug 13, 2013 9:19 PM, "Hari Shreedharan" <
>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>
>>>>>>>>>  Even though the writes are done per batch, they don't go to disk
>>>>>>>>> rightaway - commits are the only ones which actually cause an fsync - which
>>>>>>>>> is when writes actually go to disk.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Hari
>>>>>>>>>
>>>>>>>>> On Tuesday, August 13, 2013 at 7:06 PM, Pankaj Gupta wrote:
>>>>>>>>>
>>>>>>>>> Looking at the code it seems like the lock and the i/o is done per
>>>>>>>>> event and not for a batch. Is that correct? If that is the case then it
>>>>>>>>> seems like there is a lot of overhead per event. The throughput I'm seeing
>>>>>>>>> is 1 - 1.5 MBps per disk which is way below the sequential read/write
>>>>>>>>> capacity of the disk which is easily over 50MBps. Adding more sinks doesn't
>>>>>>>>> help, they just block waiting for the queue to become free. CPU usage is
>>>>>>>>> 20%, there is enough RAM for page cache so that no read is going to disk.
>>>>>>>>> The queue seems to be the bottleneck. What is the throughput I should
>>>>>>>>> expect per disk?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>>>>>
>>>>>>>>> The lock is per file. Adding more directories to the channel will
>>>>>>>>> cause more files to be created. Of course you'll need additional disks
>>>>>>>>> behind those directories to see any performance increase.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <
>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>
>>>>>>>>>  Yes, IO is done inside locks to avoid multiple takes and puts
>>>>>>>>> getting written out at the same time. Even though Java makes sure the
>>>>>>>>> writes are serialized, Flume still needs to keep track of some counters
>>>>>>>>> etc, so the lock is required. Note that the lock you are talking about is
>>>>>>>>>  in the LogFile class, which represents a single file - so even if the
>>>>>>>>> write is inside that lock (which is also inside that class itself) that
>>>>>>>>>  does not cause any contention - because the lock is just preventing two IO
>>>>>>>>> ops to happen at the same time.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Hari
>>>>>>>>>
>>>>>>>>> On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:
>>>>>>>>>
>>>>>>>>> It seems like some i/o is done inside the lock, which means that
>>>>>>>>> time for taking a lock is proportional to the time for i/o and thus it
>>>>>>>>> becomes a problem. I apologize in advance if I am wrong but the call stack
>>>>>>>>> and behavior I'm seeing seems to suggest that. Specifically, it seems that
>>>>>>>>> we do a write while inside take:
>>>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>>>>> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>         at sun.nio.ch.NativeThread.current(Native Method)
>>>>>>>>>         at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>>>>>>>>>         at
>>>>>>>>> sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>>>>>>>>>         - locked <0x00000005190ec998> (a java.lang.Object)
>>>>>>>>>         at
>>>>>>>>> org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>>>>>>>>>         at
>>>>>>>>> org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>>>>>>>>>         - locked <0x0000000519111590> (a
>>>>>>>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>>>>>>>>         at org.apache.flume.channel.file.Log.take(Log.java:550)
>>>>>>>>>         at
>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>>>>>>>>>         at
>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>>>         at
>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>>>         at
>>>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>>>         at
>>>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>>>         at
>>>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <
>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>
>>>>>>>>> Since the channel is designed to make sure that events are not
>>>>>>>>> duplicated to multiple sinks, and to protect against corruption due to
>>>>>>>>> concurrency issues, we do not need the locking in the channel's flume event
>>>>>>>>> queue. It is unlikely that locking is what is causing performance issues
>>>>>>>>> because the channel is heavily I/O bound. If you take a series of thread
>>>>>>>>> dumps, you will probably see that those threads are moving forward and the
>>>>>>>>> ones reading/writing from/to disk are the ones which are slower. These
>>>>>>>>> locks are unlikely to hit performance much.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Hari
>>>>>>>>>
>>>>>>>>> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Spent some more time debugging issues with FileChannel. The
>>>>>>>>> problem seems to lock contention reading from FlumeEventQueue:
>>>>>>>>>
>>>>>>>>> I see a lot of threads like this:
>>>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>>>>> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
>>>>>>>>> [0x00007f821afee000]
>>>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>>>         at
>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>>>>>>>>>         - waiting to lock <0x0000000518ee4c90> (a
>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>>>         at
>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>>>>>>>>>         at
>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>>>         at
>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>>>         at
>>>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>>>         at
>>>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>>>         at
>>>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have two file channels and 8 Avro Sinks per file channel. I
>>>>>>>>> added more sinks because they weren't draining fast enough. It seems like
>>>>>>>>> they send the batch then wait for an ack before sending again, thus sends
>>>>>>>>> are not pipelined and having more sinks seemed like a good way of getting
>>>>>>>>> some parallelism.
>>>>>>>>>
>>>>>>>>> Here's the full stack trace:
>>>>>>>>> 2013-08-13 15:30:32
>>>>>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02
>>>>>>>>> mixed mode):
>>>>>>>>>
>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
>>>>>>>>>    java.lang.Thread.State: WAITING (parking)
>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>> - parking to wait for  <0x000000056d932120> (a
>>>>>>>>> java.util.concurrent.CountDownLatch$Sync)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
>>>>>>>>> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
>>>>>>>>>  at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
>>>>>>>>> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
>>>>>>>>>  at
>>>>>>>>> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
>>>>>>>>> at $Proxy7.appendBatch(Unknown Source)
>>>>>>>>>  at
>>>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
>>>>>>>>> at
>>>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000 nid=0x41a1
>>>>>>>>> waiting on condition [0x0000000000000000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>
>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>>>> - parking to wait for  <0x00000005194c3340> (a
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>> - parking to wait for  <0x00000005191a16d8> (a
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>> - parking to wait for  <0x0000000519174e08> (a
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>> - parking to wait for  <0x0000000519174e98> (a
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>> - parking to wait for  <0x00000005195f9670> (a
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>> - parking to wait for  <0x0000000519174ce8> (a
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>> - parking to wait for  <0x00000005191a1918> (a
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>> - parking to wait for  <0x00000005191e7f60> (a
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153
>>>>>>>>> runnable [0x00007f8210e4d000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>  at
>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>  - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
>>>>>>>>> - locked <0x000000054a2917a8> (a
>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>  - locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>> - parking to wait for  <0x0000000519174f58> (a
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146
>>>>>>>>> runnable [0x00007f8210f4e000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>  at
>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>  - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
>>>>>>>>> - locked <0x0000000546e5dd28> (a
>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>  - locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118
>>>>>>>>> runnable [0x00007f821104f000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>  at
>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>  - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
>>>>>>>>> - locked <0x0000000544f763c0> (a
>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>  - locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111
>>>>>>>>> runnable [0x00007f8211150000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>  at
>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>  - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
>>>>>>>>> - locked <0x000000054473d5a8> (a
>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>  - locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110
>>>>>>>>> runnable [0x00007f8211251000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>  at
>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>  - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
>>>>>>>>> - locked <0x00000005437f4f70> (a
>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>  - locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f
>>>>>>>>> runnable [0x00007f8211352000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>  at
>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>  - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
>>>>>>>>> - locked <0x00000005422a9018> (a
>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>  - locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e
>>>>>>>>> runnable [0x00007f8211453000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>  at
>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>  - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
>>>>>>>>> - locked <0x0000000541063d50> (a
>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>  - locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d
>>>>>>>>> runnable [0x00007f8211554000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>  at
>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>  - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
>>>>>>>>> - locked <0x0000000540dc1110> (a
>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>  - locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a
>>>>>>>>> runnable [0x00007f8211655000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>  at
>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>  - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
>>>>>>>>> - locked <0x000000053fbbc3b0> (a
>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>  - locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109
>>>>>>>>> runnable [0x00007f8211756000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>  at
>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>  - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
>>>>>>>>> - locked <0x000000053f812860> (a
>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>  - locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd
>>>>>>>>> runnable [0x00007f8211857000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>  at
>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>  - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
>>>>>>>>> - locked <0x000000053e6e5570> (a
>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>  - locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>> - parking to wait for  <0x0000000519175138> (a
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>> - parking to wait for  <0x0000000519174c28> (a
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>> - parking to wait for  <0x00000005191751c8> (a
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0
>>>>>>>>> runnable [0x00007f8211c5b000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>  at
>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>  - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
>>>>>>>>> - locked <0x0000000536579db0> (a
>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>  - locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>> - parking to wait for  <0x00000005191750a8> (a
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1
>>>>>>>>> runnable [0x00007f8211e5d000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>  at
>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>  - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
>>>>>>>>> - locked <0x00000005313943a8> (a
>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>  - locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>> - parking to wait for  <0x000000051936f460> (a
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>> - parking to wait for  <0x000000051936dfa8> (a
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de
>>>>>>>>> runnable [0x00007f8212160000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>  at
>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>  - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
>>>>>>>>> - locked <0x000000052dad4468> (a
>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>  - locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd
>>>>>>>>> runnable [0x00007f8212261000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>  at
>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>  - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
>>>>>>>>> - locked <0x000000052c73c5f0> (a
>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>  - locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc
>>>>>>>>> runnable [0x00007f8212362000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>  at
>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>  - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
>>>>>>>>> - locked <0x000000052b8da998> (a
>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>  - locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db
>>>>>>>>> waiting for monitor entry [0x00007f8212462000]
>>>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>>> at
>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>>>>>>>>> - waiting to lock <0x0000000518ee4c90> (a
>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>>>  at
>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>>>>>>>>> at
>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>>>>>>>>  at
>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>>>>>>>> at
>>>>>>>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>>>>>>>>  at
>>>>>>>>> org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>>>>>>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>>>>>>>>  at
>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>>  at
>>>>>>>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>>>>>>>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>>>>>>>>  at
>>>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>>>  at
>>>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>>>> - parking to wait for  <0x000000051936df18> (a
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>> tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000]
>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>> - parking to wait for  <0x0000000518fe7ae8> (a
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8
>>>>>>>>> runnable [0x00007f8212766000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>  at
>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>  - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
>>>>>>>>> - locked <0x00000005263d6ba8> (a
>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>  - locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7
>>>>>>>>> runnable [0x00007f8212867000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>  at
>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>  - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
>>>>>>>>> - locked <0x0000000525d1d0e0> (a
>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>  - locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6
>>>>>>>>> runnable [0x00007f8212968000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>  at
>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>  - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
>>>>>>>>> - locked <0x0000000525a689d8> (a
>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>  - locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5
>>>>>>>>> runnable [0x00007f8212a69000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>  at
>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>  - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
>>>>>>>>> - locked <0x0000000524e1e7d8> (a
>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>  - locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>  at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>
>>>>>>>>> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4
>>>>>>>>> runnable [0x00007f8212b6a000]
>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>  at
>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>  - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
>>>>>>>>> - locked <0x0000000524d437a8> (a
>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>  - locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>  at
>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>> at
>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>
>>>>>>>>> ...
>>
>>
>
>
> --
>
>
> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com
>
> Pankaj Gupta | Software Engineer
>
> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>
>
> United States | Canada | United Kingdom | Germany
>
>
> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
> !
>



-- 
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org

Re: Lock contention in FileChannel

[Pankaj Gupta <pa...@brightroll.com>]

I did try increasing number of FileChannels. At 2 FileChannels per disk
performance seemed to be 25% better. At 4 FileChannels per disk performance
dropped to even below 1 FileChannel per disk. I will try increasing the
dataDirs tomorrow.


On Tue, Aug 13, 2013 at 8:06 PM, Brock Noland <br...@cloudera.com> wrote:

> dataDirs is a comma separated list. Try 3-4 directories and then the same
> test.
> On Aug 13, 2013 9:58 PM, "Pankaj Gupta" <pa...@brightroll.com> wrote:
>
>> Both disks were at around 15-25%.
>>
>>
>> On Tue, Aug 13, 2013 at 7:54 PM, Brock Noland <br...@cloudera.com> wrote:
>>
>>> Gotcha. When you run tge test what is tye disk utilization percentage?
>>> Iostat can be used for this.
>>> On Aug 13, 2013 9:47 PM, "Pankaj Gupta" <pa...@brightroll.com> wrote:
>>>
>>>> Those are the boxes we want to collect data from. They run flume and
>>>> send data through their avro sinks to the avro source on this box. We are
>>>> getting data at a pretty good rate and the problem is in fact that the
>>>> events don't drain from the FileChannel fast enough and the channel fill
>>>> percentage keeps getting higher.
>>>>
>>>>
>>>> On Tue, Aug 13, 2013 at 7:41 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>
>>>>> What is sending the events to the avro source?
>>>>> On Aug 13, 2013 9:34 PM, "Pankaj Gupta" <pa...@brightroll.com> wrote:
>>>>>
>>>>>> Here's the config:
>>>>>> # define channels, one for each disk
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> agent1.channels.ch1.type = FILE
>>>>>> agent1.channels.ch1.checkpointDir = /flume1/checkpoint
>>>>>> agent1.channels.ch1.dataDirs = /flume1/data
>>>>>> agent1.channels.ch1.maxFileSize = 375809638400
>>>>>> agent1.channels.ch1.capacity = 75000000
>>>>>> agent1.channels.ch1.transactionCapacity = 4000
>>>>>>
>>>>>> agent1.channels.ch2.type = FILE
>>>>>> agent1.channels.ch2.checkpointDir = /flume2/checkpoint
>>>>>> agent1.channels.ch2.dataDirs = /flume2/data
>>>>>> agent1.channels.ch2.maxFileSize = 375809638400
>>>>>> agent1.channels.ch2.capacity = 75000000
>>>>>> agent1.channels.ch2.transactionCapacity = 4000
>>>>>>
>>>>>>
>>>>>>
>>>>>> # Define an Avro source named avroSource1
>>>>>> # Each sink can connect to only one channel.
>>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>>
>>>>>>
>>>>>> agent1.sources.avroSource1.channels = ch1
>>>>>> agent1.sources.avroSource1.type = avro
>>>>>> agent1.sources.avroSource1.bind = 0.0.0.0
>>>>>> agent1.sources.avroSource1.port = <port>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> agent1.sinks.avroSink1-1-1.type = avro
>>>>>> agent1.sinks.avroSink1-1-1.channel = ch1
>>>>>> agent1.sinks.avroSink1-1-1.hostname = <hostname>
>>>>>> agent1.sinks.avroSink1-1-1.port = <port>
>>>>>> agent1.sinks.avroSink1-1-1.connect-timeout = 300000
>>>>>> agent1.sinks.avroSink1-1-1.batch-size = 4000
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> agent1.sinks.avroSink1-2-1.type = avro
>>>>>> agent1.sinks.avroSink1-2-1.channel = ch1
>>>>>> agent1.sinks.avroSink1-2-1.hostname = <hostname>
>>>>>> agent1.sinks.avroSink1-2-1.port = <port>
>>>>>> agent1.sinks.avroSink1-2-1.connect-timeout = 300000
>>>>>> agent1.sinks.avroSink1-2-1.batch-size = 4000
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> agent1.sinks.avroSink1-3-1.type = avro
>>>>>> agent1.sinks.avroSink1-3-1.channel = ch1
>>>>>> agent1.sinks.avroSink1-3-1.hostname = <hostname>
>>>>>> agent1.sinks.avroSink1-3-1.port = <port>
>>>>>> agent1.sinks.avroSink1-3-1.connect-timeout = 300000
>>>>>> agent1.sinks.avroSink1-3-1.batch-size = 4000
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> agent1.sinks.avroSink1-4-1.type = avro
>>>>>> agent1.sinks.avroSink1-4-1.channel = ch1
>>>>>> agent1.sinks.avroSink1-4-1.hostname = <hostname>
>>>>>> agent1.sinks.avroSink1-4-1.port = <port>
>>>>>> agent1.sinks.avroSink1-4-1.connect-timeout = 300000
>>>>>> agent1.sinks.avroSink1-4-1.batch-size = 4000
>>>>>>
>>>>>>
>>>>>>
>>>>>> #Add the sink groups; load-balance between each group of sinks which
>>>>>> round robin between different hops
>>>>>> agent1.sinkgroups.group1.sinks = avroSink1-1-1 avroSink1-2-1
>>>>>> avroSink1-3-1 avroSink1-4-1
>>>>>> agent1.sinkgroups.group1.processor.type = load_balance
>>>>>> agent1.sinkgroups.group1.processor.selector = ROUND_ROBIN
>>>>>> agent1.sinkgroups.group1.processor.backoff = true
>>>>>>
>>>>>>
>>>>>> #End of set
>>>>>>
>>>>>> # Define an Avro source named avroSource2
>>>>>> # Each sink can connect to only one channel.
>>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>>
>>>>>>
>>>>>> agent1.sources.avroSource2.channels = ch2
>>>>>> agent1.sources.avroSource2.type = avro
>>>>>> agent1.sources.avroSource2.bind = 0.0.0.0
>>>>>> agent1.sources.avroSource2.port = <port>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> agent1.sinks.avroSink2-1-1.type = avro
>>>>>> agent1.sinks.avroSink2-1-1.channel = ch2
>>>>>> agent1.sinks.avroSink2-1-1.hostname = <hostname>
>>>>>> agent1.sinks.avroSink2-1-1.port = <port>
>>>>>> agent1.sinks.avroSink2-1-1.connect-timeout = 300000
>>>>>> agent1.sinks.avroSink2-1-1.batch-size = 4000
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> agent1.sinks.avroSink2-2-1.type = avro
>>>>>> agent1.sinks.avroSink2-2-1.channel = ch2
>>>>>> agent1.sinks.avroSink2-2-1.hostname = <hostname>
>>>>>> agent1.sinks.avroSink2-2-1.port = <port>
>>>>>> agent1.sinks.avroSink2-2-1.connect-timeout = 300000
>>>>>> agent1.sinks.avroSink2-2-1.batch-size = 4000
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> agent1.sinks.avroSink2-3-1.type = avro
>>>>>> agent1.sinks.avroSink2-3-1.channel = ch2
>>>>>> agent1.sinks.avroSink2-3-1.hostname = <hostname>
>>>>>> agent1.sinks.avroSink2-3-1.port = <port>
>>>>>> agent1.sinks.avroSink2-3-1.connect-timeout = 300000
>>>>>> agent1.sinks.avroSink2-3-1.batch-size = 4000
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> agent1.sinks.avroSink2-4-1.type = avro
>>>>>> agent1.sinks.avroSink2-4-1.channel = ch2
>>>>>> agent1.sinks.avroSink2-4-1.hostname = <hostname>
>>>>>> agent1.sinks.avroSink2-4-1.port = <port>
>>>>>> agent1.sinks.avroSink2-4-1.connect-timeout = 300000
>>>>>> agent1.sinks.avroSink2-4-1.batch-size = 4000
>>>>>>
>>>>>>
>>>>>>
>>>>>> #Add the sink groups; load-balance between each group of sinks which
>>>>>> round robin between different hops
>>>>>> agent1.sinkgroups.group2.sinks = avroSink2-1-1 avroSink2-2-1
>>>>>> avroSink2-3-1 avroSink2-4-1
>>>>>> agent1.sinkgroups.group2.processor.type = load_balance
>>>>>> agent1.sinkgroups.group2.processor.selector = ROUND_ROBIN
>>>>>> agent1.sinkgroups.group2.processor.backoff = true
>>>>>>
>>>>>>
>>>>>> #End of set
>>>>>>
>>>>>> # Define an Avro source named avroSource3
>>>>>> # Each sink can connect to only one channel.
>>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>>
>>>>>>
>>>>>> agent1.sources.avroSource3.channels = ch1
>>>>>> agent1.sources.avroSource3.type = avro
>>>>>> agent1.sources.avroSource3.bind = 0.0.0.0
>>>>>> agent1.sources.avroSource3.port = <port>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> agent1.sinks.avroSink3-1-1.type = avro
>>>>>> agent1.sinks.avroSink3-1-1.channel = ch1
>>>>>> agent1.sinks.avroSink3-1-1.hostname = <hostname>
>>>>>> agent1.sinks.avroSink3-1-1.port = <port>
>>>>>> agent1.sinks.avroSink3-1-1.connect-timeout = 300000
>>>>>> agent1.sinks.avroSink3-1-1.batch-size = 4000
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> agent1.sinks.avroSink3-2-1.type = avro
>>>>>> agent1.sinks.avroSink3-2-1.channel = ch1
>>>>>> agent1.sinks.avroSink3-2-1.hostname = <hostname>
>>>>>> agent1.sinks.avroSink3-2-1.port = <port>
>>>>>> agent1.sinks.avroSink3-2-1.connect-timeout = 300000
>>>>>> agent1.sinks.avroSink3-2-1.batch-size = 4000
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> agent1.sinks.avroSink3-3-1.type = avro
>>>>>> agent1.sinks.avroSink3-3-1.channel = ch1
>>>>>> agent1.sinks.avroSink3-3-1.hostname = <hostname>
>>>>>> agent1.sinks.avroSink3-3-1.port = <port>
>>>>>> agent1.sinks.avroSink3-3-1.connect-timeout = 300000
>>>>>> agent1.sinks.avroSink3-3-1.batch-size = 4000
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> agent1.sinks.avroSink3-4-1.type = avro
>>>>>> agent1.sinks.avroSink3-4-1.channel = ch1
>>>>>> agent1.sinks.avroSink3-4-1.hostname = <hostname>
>>>>>> agent1.sinks.avroSink3-4-1.port = <port>
>>>>>> agent1.sinks.avroSink3-4-1.connect-timeout = 300000
>>>>>> agent1.sinks.avroSink3-4-1.batch-size = 4000
>>>>>>
>>>>>>
>>>>>>
>>>>>> #Add the sink groups; load-balance between each group of sinks which
>>>>>> round robin between different hops
>>>>>> agent1.sinkgroups.group3.sinks = avroSink3-1-1 avroSink3-2-1
>>>>>> avroSink3-3-1 avroSink3-4-1
>>>>>> agent1.sinkgroups.group3.processor.type = load_balance
>>>>>> agent1.sinkgroups.group3.processor.selector = ROUND_ROBIN
>>>>>> agent1.sinkgroups.group3.processor.backoff = true
>>>>>>
>>>>>>
>>>>>> #End of set
>>>>>>
>>>>>> # Define an Avro source named avroSource4
>>>>>> # Each sink can connect to only one channel.
>>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>>
>>>>>>
>>>>>> agent1.sources.avroSource4.channels = ch2
>>>>>> agent1.sources.avroSource4.type = avro
>>>>>> agent1.sources.avroSource4.bind = 0.0.0.0
>>>>>> agent1.sources.avroSource4.port = <port>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> agent1.sinks.avroSink4-1-1.type = avro
>>>>>> agent1.sinks.avroSink4-1-1.channel = ch2
>>>>>> agent1.sinks.avroSink4-1-1.hostname = <hostname>
>>>>>> agent1.sinks.avroSink4-1-1.port = <port>
>>>>>> agent1.sinks.avroSink4-1-1.connect-timeout = 300000
>>>>>> agent1.sinks.avroSink4-1-1.batch-size = 4000
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> agent1.sinks.avroSink4-2-1.type = avro
>>>>>> agent1.sinks.avroSink4-2-1.channel = ch2
>>>>>> agent1.sinks.avroSink4-2-1.hostname = <hostname>
>>>>>> agent1.sinks.avroSink4-2-1.port = <port>
>>>>>> agent1.sinks.avroSink4-2-1.connect-timeout = 300000
>>>>>> agent1.sinks.avroSink4-2-1.batch-size = 4000
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> agent1.sinks.avroSink4-3-1.type = avro
>>>>>> agent1.sinks.avroSink4-3-1.channel = ch2
>>>>>> agent1.sinks.avroSink4-3-1.hostname = <hostname>
>>>>>> agent1.sinks.avroSink4-3-1.port = <port>
>>>>>> agent1.sinks.avroSink4-3-1.connect-timeout = 300000
>>>>>> agent1.sinks.avroSink4-3-1.batch-size = 4000
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> agent1.sinks.avroSink4-4-1.type = avro
>>>>>> agent1.sinks.avroSink4-4-1.channel = ch2
>>>>>> agent1.sinks.avroSink4-4-1.hostname = <hostname>
>>>>>> agent1.sinks.avroSink4-4-1.port = <port>
>>>>>> agent1.sinks.avroSink4-4-1.connect-timeout = 300000
>>>>>> agent1.sinks.avroSink4-4-1.batch-size = 4000
>>>>>>
>>>>>>
>>>>>>
>>>>>> #Add the sink groups; load-balance between each group of sinks which
>>>>>> round robin between different hops
>>>>>> agent1.sinkgroups.group4.sinks = avroSink4-1-1 avroSink4-2-1
>>>>>> avroSink4-3-1 avroSink4-4-1
>>>>>> agent1.sinkgroups.group4.processor.type = load_balance
>>>>>> agent1.sinkgroups.group4.processor.selector = ROUND_ROBIN
>>>>>> agent1.sinkgroups.group4.processor.backoff = true
>>>>>>
>>>>>>
>>>>>> #End of set
>>>>>>
>>>>>>
>>>>>>
>>>>>> # Finally, now that we've defined all of our components, tell
>>>>>> # agent1 which ones we want to activate.
>>>>>> agent1.channels = ch1 ch2
>>>>>> agent1.sources = avroSource1 avroSource2 avroSource3 avroSource4
>>>>>> agent1.sinks = avroSink1-1-1 avroSink1-2-1 avroSink1-3-1
>>>>>> avroSink1-4-1 avroSink2-1-1 avroSink2-2-1 avroSink2-3-1 avroSink2-4-1
>>>>>> avroSink3-1-1 avroSink3-2-1 avroSink3-3-1 avroSink3-4-1 avroSink4-1-1
>>>>>> avroSink4-2-1 avroSink4-3-1 avroSink4-4-1
>>>>>> agent1.sinkgroups = group1 group2 group3 group4
>>>>>>
>>>>>> There are two disks on the box /flume1 and /flume2
>>>>>> 32 cores of CPU, 32 GB RAM, 1GbPS ethernet.
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 13, 2013 at 7:22 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>>
>>>>>>> Can you share your conf file?
>>>>>>> On Aug 13, 2013 9:19 PM, "Hari Shreedharan" <
>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>
>>>>>>>>  Even though the writes are done per batch, they don't go to disk
>>>>>>>> rightaway - commits are the only ones which actually cause an fsync - which
>>>>>>>> is when writes actually go to disk.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Hari
>>>>>>>>
>>>>>>>> On Tuesday, August 13, 2013 at 7:06 PM, Pankaj Gupta wrote:
>>>>>>>>
>>>>>>>> Looking at the code it seems like the lock and the i/o is done per
>>>>>>>> event and not for a batch. Is that correct? If that is the case then it
>>>>>>>> seems like there is a lot of overhead per event. The throughput I'm seeing
>>>>>>>> is 1 - 1.5 MBps per disk which is way below the sequential read/write
>>>>>>>> capacity of the disk which is easily over 50MBps. Adding more sinks doesn't
>>>>>>>> help, they just block waiting for the queue to become free. CPU usage is
>>>>>>>> 20%, there is enough RAM for page cache so that no read is going to disk.
>>>>>>>> The queue seems to be the bottleneck. What is the throughput I should
>>>>>>>> expect per disk?
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>>>>
>>>>>>>> The lock is per file. Adding more directories to the channel will
>>>>>>>> cause more files to be created. Of course you'll need additional disks
>>>>>>>> behind those directories to see any performance increase.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <
>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>
>>>>>>>>  Yes, IO is done inside locks to avoid multiple takes and puts
>>>>>>>> getting written out at the same time. Even though Java makes sure the
>>>>>>>> writes are serialized, Flume still needs to keep track of some counters
>>>>>>>> etc, so the lock is required. Note that the lock you are talking about is
>>>>>>>>  in the LogFile class, which represents a single file - so even if the
>>>>>>>> write is inside that lock (which is also inside that class itself) that
>>>>>>>>  does not cause any contention - because the lock is just preventing two IO
>>>>>>>> ops to happen at the same time.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Hari
>>>>>>>>
>>>>>>>> On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:
>>>>>>>>
>>>>>>>> It seems like some i/o is done inside the lock, which means that
>>>>>>>> time for taking a lock is proportional to the time for i/o and thus it
>>>>>>>> becomes a problem. I apologize in advance if I am wrong but the call stack
>>>>>>>> and behavior I'm seeing seems to suggest that. Specifically, it seems that
>>>>>>>> we do a write while inside take:
>>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>>>> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>         at sun.nio.ch.NativeThread.current(Native Method)
>>>>>>>>         at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>>>>>>>>         at
>>>>>>>> sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>>>>>>>>         - locked <0x00000005190ec998> (a java.lang.Object)
>>>>>>>>         at
>>>>>>>> org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>>>>>>>>         at
>>>>>>>> org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>>>>>>>>         - locked <0x0000000519111590> (a
>>>>>>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>>>>>>>         at org.apache.flume.channel.file.Log.take(Log.java:550)
>>>>>>>>         at
>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>>>>>>>>         at
>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>>         at
>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>>         at
>>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>>         at
>>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>>         at
>>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <
>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>
>>>>>>>> Since the channel is designed to make sure that events are not
>>>>>>>> duplicated to multiple sinks, and to protect against corruption due to
>>>>>>>> concurrency issues, we do not need the locking in the channel's flume event
>>>>>>>> queue. It is unlikely that locking is what is causing performance issues
>>>>>>>> because the channel is heavily I/O bound. If you take a series of thread
>>>>>>>> dumps, you will probably see that those threads are moving forward and the
>>>>>>>> ones reading/writing from/to disk are the ones which are slower. These
>>>>>>>> locks are unlikely to hit performance much.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Hari
>>>>>>>>
>>>>>>>> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Spent some more time debugging issues with FileChannel. The problem
>>>>>>>> seems to lock contention reading from FlumeEventQueue:
>>>>>>>>
>>>>>>>> I see a lot of threads like this:
>>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>>>> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
>>>>>>>> [0x00007f821afee000]
>>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>>         at
>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>>>>>>>>         - waiting to lock <0x0000000518ee4c90> (a
>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>>         at
>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>>>>>>>>         at
>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>>         at
>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>>         at
>>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>>         at
>>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>>         at
>>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>>
>>>>>>>> I have two file channels and 8 Avro Sinks per file channel. I added
>>>>>>>> more sinks because they weren't draining fast enough. It seems like they
>>>>>>>> send the batch then wait for an ack before sending again, thus sends are
>>>>>>>> not pipelined and having more sinks seemed like a good way of getting some
>>>>>>>> parallelism.
>>>>>>>>
>>>>>>>> Here's the full stack trace:
>>>>>>>> 2013-08-13 15:30:32
>>>>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02 mixed
>>>>>>>> mode):
>>>>>>>>
>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
>>>>>>>>    java.lang.Thread.State: WAITING (parking)
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> - parking to wait for  <0x000000056d932120> (a
>>>>>>>> java.util.concurrent.CountDownLatch$Sync)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>>>>>>>> at
>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>>>>>>>> at
>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
>>>>>>>> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
>>>>>>>>  at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
>>>>>>>> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
>>>>>>>>  at
>>>>>>>> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
>>>>>>>> at $Proxy7.appendBatch(Unknown Source)
>>>>>>>>  at
>>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
>>>>>>>> at
>>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000 nid=0x41a1
>>>>>>>> waiting on condition [0x0000000000000000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>
>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>>> - parking to wait for  <0x00000005194c3340> (a
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> - parking to wait for  <0x00000005191a16d8> (a
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> - parking to wait for  <0x0000000519174e08> (a
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> - parking to wait for  <0x0000000519174e98> (a
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> - parking to wait for  <0x00000005195f9670> (a
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> - parking to wait for  <0x0000000519174ce8> (a
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> - parking to wait for  <0x00000005191a1918> (a
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> - parking to wait for  <0x00000005191e7f60> (a
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153
>>>>>>>> runnable [0x00007f8210e4d000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>  at
>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>  - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
>>>>>>>> - locked <0x000000054a2917a8> (a
>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>  - locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>> at
>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> - parking to wait for  <0x0000000519174f58> (a
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146
>>>>>>>> runnable [0x00007f8210f4e000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>  at
>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>  - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
>>>>>>>> - locked <0x0000000546e5dd28> (a
>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>  - locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>> at
>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118
>>>>>>>> runnable [0x00007f821104f000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>  at
>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>  - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
>>>>>>>> - locked <0x0000000544f763c0> (a
>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>  - locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>> at
>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111
>>>>>>>> runnable [0x00007f8211150000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>  at
>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>  - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
>>>>>>>> - locked <0x000000054473d5a8> (a
>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>  - locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>> at
>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110
>>>>>>>> runnable [0x00007f8211251000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>  at
>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>  - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
>>>>>>>> - locked <0x00000005437f4f70> (a
>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>  - locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>> at
>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f
>>>>>>>> runnable [0x00007f8211352000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>  at
>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>  - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
>>>>>>>> - locked <0x00000005422a9018> (a
>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>  - locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>> at
>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e
>>>>>>>> runnable [0x00007f8211453000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>  at
>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>  - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
>>>>>>>> - locked <0x0000000541063d50> (a
>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>  - locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>> at
>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d
>>>>>>>> runnable [0x00007f8211554000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>  at
>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>  - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
>>>>>>>> - locked <0x0000000540dc1110> (a
>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>  - locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>> at
>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a
>>>>>>>> runnable [0x00007f8211655000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>  at
>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>  - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
>>>>>>>> - locked <0x000000053fbbc3b0> (a
>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>  - locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>> at
>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109
>>>>>>>> runnable [0x00007f8211756000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>  at
>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>  - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
>>>>>>>> - locked <0x000000053f812860> (a
>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>  - locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>> at
>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd
>>>>>>>> runnable [0x00007f8211857000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>  at
>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>  - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
>>>>>>>> - locked <0x000000053e6e5570> (a
>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>  - locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>> at
>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> - parking to wait for  <0x0000000519175138> (a
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> - parking to wait for  <0x0000000519174c28> (a
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> - parking to wait for  <0x00000005191751c8> (a
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0
>>>>>>>> runnable [0x00007f8211c5b000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>  at
>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>  - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
>>>>>>>> - locked <0x0000000536579db0> (a
>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>  - locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>> at
>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> - parking to wait for  <0x00000005191750a8> (a
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1
>>>>>>>> runnable [0x00007f8211e5d000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>  at
>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>  - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
>>>>>>>> - locked <0x00000005313943a8> (a
>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>  - locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>> at
>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> - parking to wait for  <0x000000051936f460> (a
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> - parking to wait for  <0x000000051936dfa8> (a
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de
>>>>>>>> runnable [0x00007f8212160000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>  at
>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>  - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
>>>>>>>> - locked <0x000000052dad4468> (a
>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>  - locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>> at
>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd
>>>>>>>> runnable [0x00007f8212261000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>  at
>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>  - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
>>>>>>>> - locked <0x000000052c73c5f0> (a
>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>  - locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>> at
>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc
>>>>>>>> runnable [0x00007f8212362000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>  at
>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>  - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
>>>>>>>> - locked <0x000000052b8da998> (a
>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>  - locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>> at
>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db
>>>>>>>> waiting for monitor entry [0x00007f8212462000]
>>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>> at
>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>>>>>>>> - waiting to lock <0x0000000518ee4c90> (a
>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>>  at
>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>>>>>>>> at
>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>>>>>>>  at
>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>>>>>>> at
>>>>>>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>>>>>>>  at
>>>>>>>> org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>>>>>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>>>>>>>  at
>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>  at
>>>>>>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>>>>>>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>>>>>>>  at
>>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>>  at
>>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>>>>>>> at
>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>>> - parking to wait for  <0x000000051936df18> (a
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>> tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000]
>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> - parking to wait for  <0x0000000518fe7ae8> (a
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8
>>>>>>>> runnable [0x00007f8212766000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>  at
>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>  - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
>>>>>>>> - locked <0x00000005263d6ba8> (a
>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>  - locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>> at
>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7
>>>>>>>> runnable [0x00007f8212867000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>  at
>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>  - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
>>>>>>>> - locked <0x0000000525d1d0e0> (a
>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>  - locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>> at
>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6
>>>>>>>> runnable [0x00007f8212968000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>  at
>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>  - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
>>>>>>>> - locked <0x0000000525a689d8> (a
>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>  - locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>> at
>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5
>>>>>>>> runnable [0x00007f8212a69000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>  at
>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>  - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
>>>>>>>> - locked <0x0000000524e1e7d8> (a
>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>  - locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>> at
>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>  at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>
>>>>>>>> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4
>>>>>>>> runnable [0x00007f8212b6a000]
>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>  at
>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>  - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
>>>>>>>> - locked <0x0000000524d437a8> (a
>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>  - locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>> at
>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>  at
>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>> at
>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>
>>>>>>>> ...
>
>


-- 


*P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com

Pankaj Gupta | Software Engineer

*BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com


United States | Canada | United Kingdom | Germany


We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
!

Re: Lock contention in FileChannel

[Brock Noland <br...@cloudera.com>]

dataDirs is a comma separated list. Try 3-4 directories and then the same
test.
On Aug 13, 2013 9:58 PM, "Pankaj Gupta" <pa...@brightroll.com> wrote:

> Both disks were at around 15-25%.
>
>
> On Tue, Aug 13, 2013 at 7:54 PM, Brock Noland <br...@cloudera.com> wrote:
>
>> Gotcha. When you run tge test what is tye disk utilization percentage?
>> Iostat can be used for this.
>> On Aug 13, 2013 9:47 PM, "Pankaj Gupta" <pa...@brightroll.com> wrote:
>>
>>> Those are the boxes we want to collect data from. They run flume and
>>> send data through their avro sinks to the avro source on this box. We are
>>> getting data at a pretty good rate and the problem is in fact that the
>>> events don't drain from the FileChannel fast enough and the channel fill
>>> percentage keeps getting higher.
>>>
>>>
>>> On Tue, Aug 13, 2013 at 7:41 PM, Brock Noland <br...@cloudera.com>wrote:
>>>
>>>> What is sending the events to the avro source?
>>>> On Aug 13, 2013 9:34 PM, "Pankaj Gupta" <pa...@brightroll.com> wrote:
>>>>
>>>>> Here's the config:
>>>>> # define channels, one for each disk
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> agent1.channels.ch1.type = FILE
>>>>> agent1.channels.ch1.checkpointDir = /flume1/checkpoint
>>>>> agent1.channels.ch1.dataDirs = /flume1/data
>>>>> agent1.channels.ch1.maxFileSize = 375809638400
>>>>> agent1.channels.ch1.capacity = 75000000
>>>>> agent1.channels.ch1.transactionCapacity = 4000
>>>>>
>>>>> agent1.channels.ch2.type = FILE
>>>>> agent1.channels.ch2.checkpointDir = /flume2/checkpoint
>>>>> agent1.channels.ch2.dataDirs = /flume2/data
>>>>> agent1.channels.ch2.maxFileSize = 375809638400
>>>>> agent1.channels.ch2.capacity = 75000000
>>>>> agent1.channels.ch2.transactionCapacity = 4000
>>>>>
>>>>>
>>>>>
>>>>> # Define an Avro source named avroSource1
>>>>> # Each sink can connect to only one channel.
>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>
>>>>>
>>>>> agent1.sources.avroSource1.channels = ch1
>>>>> agent1.sources.avroSource1.type = avro
>>>>> agent1.sources.avroSource1.bind = 0.0.0.0
>>>>> agent1.sources.avroSource1.port = <port>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> agent1.sinks.avroSink1-1-1.type = avro
>>>>> agent1.sinks.avroSink1-1-1.channel = ch1
>>>>> agent1.sinks.avroSink1-1-1.hostname = <hostname>
>>>>> agent1.sinks.avroSink1-1-1.port = <port>
>>>>> agent1.sinks.avroSink1-1-1.connect-timeout = 300000
>>>>> agent1.sinks.avroSink1-1-1.batch-size = 4000
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> agent1.sinks.avroSink1-2-1.type = avro
>>>>> agent1.sinks.avroSink1-2-1.channel = ch1
>>>>> agent1.sinks.avroSink1-2-1.hostname = <hostname>
>>>>> agent1.sinks.avroSink1-2-1.port = <port>
>>>>> agent1.sinks.avroSink1-2-1.connect-timeout = 300000
>>>>> agent1.sinks.avroSink1-2-1.batch-size = 4000
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> agent1.sinks.avroSink1-3-1.type = avro
>>>>> agent1.sinks.avroSink1-3-1.channel = ch1
>>>>> agent1.sinks.avroSink1-3-1.hostname = <hostname>
>>>>> agent1.sinks.avroSink1-3-1.port = <port>
>>>>> agent1.sinks.avroSink1-3-1.connect-timeout = 300000
>>>>> agent1.sinks.avroSink1-3-1.batch-size = 4000
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> agent1.sinks.avroSink1-4-1.type = avro
>>>>> agent1.sinks.avroSink1-4-1.channel = ch1
>>>>> agent1.sinks.avroSink1-4-1.hostname = <hostname>
>>>>> agent1.sinks.avroSink1-4-1.port = <port>
>>>>> agent1.sinks.avroSink1-4-1.connect-timeout = 300000
>>>>> agent1.sinks.avroSink1-4-1.batch-size = 4000
>>>>>
>>>>>
>>>>>
>>>>> #Add the sink groups; load-balance between each group of sinks which
>>>>> round robin between different hops
>>>>> agent1.sinkgroups.group1.sinks = avroSink1-1-1 avroSink1-2-1
>>>>> avroSink1-3-1 avroSink1-4-1
>>>>> agent1.sinkgroups.group1.processor.type = load_balance
>>>>> agent1.sinkgroups.group1.processor.selector = ROUND_ROBIN
>>>>> agent1.sinkgroups.group1.processor.backoff = true
>>>>>
>>>>>
>>>>> #End of set
>>>>>
>>>>> # Define an Avro source named avroSource2
>>>>> # Each sink can connect to only one channel.
>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>
>>>>>
>>>>> agent1.sources.avroSource2.channels = ch2
>>>>> agent1.sources.avroSource2.type = avro
>>>>> agent1.sources.avroSource2.bind = 0.0.0.0
>>>>> agent1.sources.avroSource2.port = <port>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> agent1.sinks.avroSink2-1-1.type = avro
>>>>> agent1.sinks.avroSink2-1-1.channel = ch2
>>>>> agent1.sinks.avroSink2-1-1.hostname = <hostname>
>>>>> agent1.sinks.avroSink2-1-1.port = <port>
>>>>> agent1.sinks.avroSink2-1-1.connect-timeout = 300000
>>>>> agent1.sinks.avroSink2-1-1.batch-size = 4000
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> agent1.sinks.avroSink2-2-1.type = avro
>>>>> agent1.sinks.avroSink2-2-1.channel = ch2
>>>>> agent1.sinks.avroSink2-2-1.hostname = <hostname>
>>>>> agent1.sinks.avroSink2-2-1.port = <port>
>>>>> agent1.sinks.avroSink2-2-1.connect-timeout = 300000
>>>>> agent1.sinks.avroSink2-2-1.batch-size = 4000
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> agent1.sinks.avroSink2-3-1.type = avro
>>>>> agent1.sinks.avroSink2-3-1.channel = ch2
>>>>> agent1.sinks.avroSink2-3-1.hostname = <hostname>
>>>>> agent1.sinks.avroSink2-3-1.port = <port>
>>>>> agent1.sinks.avroSink2-3-1.connect-timeout = 300000
>>>>> agent1.sinks.avroSink2-3-1.batch-size = 4000
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> agent1.sinks.avroSink2-4-1.type = avro
>>>>> agent1.sinks.avroSink2-4-1.channel = ch2
>>>>> agent1.sinks.avroSink2-4-1.hostname = <hostname>
>>>>> agent1.sinks.avroSink2-4-1.port = <port>
>>>>> agent1.sinks.avroSink2-4-1.connect-timeout = 300000
>>>>> agent1.sinks.avroSink2-4-1.batch-size = 4000
>>>>>
>>>>>
>>>>>
>>>>> #Add the sink groups; load-balance between each group of sinks which
>>>>> round robin between different hops
>>>>> agent1.sinkgroups.group2.sinks = avroSink2-1-1 avroSink2-2-1
>>>>> avroSink2-3-1 avroSink2-4-1
>>>>> agent1.sinkgroups.group2.processor.type = load_balance
>>>>> agent1.sinkgroups.group2.processor.selector = ROUND_ROBIN
>>>>> agent1.sinkgroups.group2.processor.backoff = true
>>>>>
>>>>>
>>>>> #End of set
>>>>>
>>>>> # Define an Avro source named avroSource3
>>>>> # Each sink can connect to only one channel.
>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>
>>>>>
>>>>> agent1.sources.avroSource3.channels = ch1
>>>>> agent1.sources.avroSource3.type = avro
>>>>> agent1.sources.avroSource3.bind = 0.0.0.0
>>>>> agent1.sources.avroSource3.port = <port>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> agent1.sinks.avroSink3-1-1.type = avro
>>>>> agent1.sinks.avroSink3-1-1.channel = ch1
>>>>> agent1.sinks.avroSink3-1-1.hostname = <hostname>
>>>>> agent1.sinks.avroSink3-1-1.port = <port>
>>>>> agent1.sinks.avroSink3-1-1.connect-timeout = 300000
>>>>> agent1.sinks.avroSink3-1-1.batch-size = 4000
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> agent1.sinks.avroSink3-2-1.type = avro
>>>>> agent1.sinks.avroSink3-2-1.channel = ch1
>>>>> agent1.sinks.avroSink3-2-1.hostname = <hostname>
>>>>> agent1.sinks.avroSink3-2-1.port = <port>
>>>>> agent1.sinks.avroSink3-2-1.connect-timeout = 300000
>>>>> agent1.sinks.avroSink3-2-1.batch-size = 4000
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> agent1.sinks.avroSink3-3-1.type = avro
>>>>> agent1.sinks.avroSink3-3-1.channel = ch1
>>>>> agent1.sinks.avroSink3-3-1.hostname = <hostname>
>>>>> agent1.sinks.avroSink3-3-1.port = <port>
>>>>> agent1.sinks.avroSink3-3-1.connect-timeout = 300000
>>>>> agent1.sinks.avroSink3-3-1.batch-size = 4000
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> agent1.sinks.avroSink3-4-1.type = avro
>>>>> agent1.sinks.avroSink3-4-1.channel = ch1
>>>>> agent1.sinks.avroSink3-4-1.hostname = <hostname>
>>>>> agent1.sinks.avroSink3-4-1.port = <port>
>>>>> agent1.sinks.avroSink3-4-1.connect-timeout = 300000
>>>>> agent1.sinks.avroSink3-4-1.batch-size = 4000
>>>>>
>>>>>
>>>>>
>>>>> #Add the sink groups; load-balance between each group of sinks which
>>>>> round robin between different hops
>>>>> agent1.sinkgroups.group3.sinks = avroSink3-1-1 avroSink3-2-1
>>>>> avroSink3-3-1 avroSink3-4-1
>>>>> agent1.sinkgroups.group3.processor.type = load_balance
>>>>> agent1.sinkgroups.group3.processor.selector = ROUND_ROBIN
>>>>> agent1.sinkgroups.group3.processor.backoff = true
>>>>>
>>>>>
>>>>> #End of set
>>>>>
>>>>> # Define an Avro source named avroSource4
>>>>> # Each sink can connect to only one channel.
>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>
>>>>>
>>>>> agent1.sources.avroSource4.channels = ch2
>>>>> agent1.sources.avroSource4.type = avro
>>>>> agent1.sources.avroSource4.bind = 0.0.0.0
>>>>> agent1.sources.avroSource4.port = <port>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> agent1.sinks.avroSink4-1-1.type = avro
>>>>> agent1.sinks.avroSink4-1-1.channel = ch2
>>>>> agent1.sinks.avroSink4-1-1.hostname = <hostname>
>>>>> agent1.sinks.avroSink4-1-1.port = <port>
>>>>> agent1.sinks.avroSink4-1-1.connect-timeout = 300000
>>>>> agent1.sinks.avroSink4-1-1.batch-size = 4000
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> agent1.sinks.avroSink4-2-1.type = avro
>>>>> agent1.sinks.avroSink4-2-1.channel = ch2
>>>>> agent1.sinks.avroSink4-2-1.hostname = <hostname>
>>>>> agent1.sinks.avroSink4-2-1.port = <port>
>>>>> agent1.sinks.avroSink4-2-1.connect-timeout = 300000
>>>>> agent1.sinks.avroSink4-2-1.batch-size = 4000
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> agent1.sinks.avroSink4-3-1.type = avro
>>>>> agent1.sinks.avroSink4-3-1.channel = ch2
>>>>> agent1.sinks.avroSink4-3-1.hostname = <hostname>
>>>>> agent1.sinks.avroSink4-3-1.port = <port>
>>>>> agent1.sinks.avroSink4-3-1.connect-timeout = 300000
>>>>> agent1.sinks.avroSink4-3-1.batch-size = 4000
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> agent1.sinks.avroSink4-4-1.type = avro
>>>>> agent1.sinks.avroSink4-4-1.channel = ch2
>>>>> agent1.sinks.avroSink4-4-1.hostname = <hostname>
>>>>> agent1.sinks.avroSink4-4-1.port = <port>
>>>>> agent1.sinks.avroSink4-4-1.connect-timeout = 300000
>>>>> agent1.sinks.avroSink4-4-1.batch-size = 4000
>>>>>
>>>>>
>>>>>
>>>>> #Add the sink groups; load-balance between each group of sinks which
>>>>> round robin between different hops
>>>>> agent1.sinkgroups.group4.sinks = avroSink4-1-1 avroSink4-2-1
>>>>> avroSink4-3-1 avroSink4-4-1
>>>>> agent1.sinkgroups.group4.processor.type = load_balance
>>>>> agent1.sinkgroups.group4.processor.selector = ROUND_ROBIN
>>>>> agent1.sinkgroups.group4.processor.backoff = true
>>>>>
>>>>>
>>>>> #End of set
>>>>>
>>>>>
>>>>>
>>>>> # Finally, now that we've defined all of our components, tell
>>>>> # agent1 which ones we want to activate.
>>>>> agent1.channels = ch1 ch2
>>>>> agent1.sources = avroSource1 avroSource2 avroSource3 avroSource4
>>>>> agent1.sinks = avroSink1-1-1 avroSink1-2-1 avroSink1-3-1 avroSink1-4-1
>>>>> avroSink2-1-1 avroSink2-2-1 avroSink2-3-1 avroSink2-4-1 avroSink3-1-1
>>>>> avroSink3-2-1 avroSink3-3-1 avroSink3-4-1 avroSink4-1-1 avroSink4-2-1
>>>>> avroSink4-3-1 avroSink4-4-1
>>>>> agent1.sinkgroups = group1 group2 group3 group4
>>>>>
>>>>> There are two disks on the box /flume1 and /flume2
>>>>> 32 cores of CPU, 32 GB RAM, 1GbPS ethernet.
>>>>>
>>>>>
>>>>> On Tue, Aug 13, 2013 at 7:22 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>
>>>>>> Can you share your conf file?
>>>>>> On Aug 13, 2013 9:19 PM, "Hari Shreedharan" <
>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>
>>>>>>>  Even though the writes are done per batch, they don't go to disk
>>>>>>> rightaway - commits are the only ones which actually cause an fsync - which
>>>>>>> is when writes actually go to disk.
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Hari
>>>>>>>
>>>>>>> On Tuesday, August 13, 2013 at 7:06 PM, Pankaj Gupta wrote:
>>>>>>>
>>>>>>> Looking at the code it seems like the lock and the i/o is done per
>>>>>>> event and not for a batch. Is that correct? If that is the case then it
>>>>>>> seems like there is a lot of overhead per event. The throughput I'm seeing
>>>>>>> is 1 - 1.5 MBps per disk which is way below the sequential read/write
>>>>>>> capacity of the disk which is easily over 50MBps. Adding more sinks doesn't
>>>>>>> help, they just block waiting for the queue to become free. CPU usage is
>>>>>>> 20%, there is enough RAM for page cache so that no read is going to disk.
>>>>>>> The queue seems to be the bottleneck. What is the throughput I should
>>>>>>> expect per disk?
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>>>
>>>>>>> The lock is per file. Adding more directories to the channel will
>>>>>>> cause more files to be created. Of course you'll need additional disks
>>>>>>> behind those directories to see any performance increase.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <
>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>
>>>>>>>  Yes, IO is done inside locks to avoid multiple takes and puts
>>>>>>> getting written out at the same time. Even though Java makes sure the
>>>>>>> writes are serialized, Flume still needs to keep track of some counters
>>>>>>> etc, so the lock is required. Note that the lock you are talking about is
>>>>>>>  in the LogFile class, which represents a single file - so even if the
>>>>>>> write is inside that lock (which is also inside that class itself) that
>>>>>>>  does not cause any contention - because the lock is just preventing two IO
>>>>>>> ops to happen at the same time.
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Hari
>>>>>>>
>>>>>>> On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:
>>>>>>>
>>>>>>> It seems like some i/o is done inside the lock, which means that
>>>>>>> time for taking a lock is proportional to the time for i/o and thus it
>>>>>>> becomes a problem. I apologize in advance if I am wrong but the call stack
>>>>>>> and behavior I'm seeing seems to suggest that. Specifically, it seems that
>>>>>>> we do a write while inside take:
>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>>> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>         at sun.nio.ch.NativeThread.current(Native Method)
>>>>>>>         at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>>>>>>>         at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>>>>>>>         - locked <0x00000005190ec998> (a java.lang.Object)
>>>>>>>         at
>>>>>>> org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>>>>>>>         at
>>>>>>> org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>>>>>>>         - locked <0x0000000519111590> (a
>>>>>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>>>>>>         at org.apache.flume.channel.file.Log.take(Log.java:550)
>>>>>>>         at
>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>>>>>>>         at
>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>         at
>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>         at
>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>         at
>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>         at
>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <
>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>
>>>>>>> Since the channel is designed to make sure that events are not
>>>>>>> duplicated to multiple sinks, and to protect against corruption due to
>>>>>>> concurrency issues, we do not need the locking in the channel's flume event
>>>>>>> queue. It is unlikely that locking is what is causing performance issues
>>>>>>> because the channel is heavily I/O bound. If you take a series of thread
>>>>>>> dumps, you will probably see that those threads are moving forward and the
>>>>>>> ones reading/writing from/to disk are the ones which are slower. These
>>>>>>> locks are unlikely to hit performance much.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Hari
>>>>>>>
>>>>>>> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Spent some more time debugging issues with FileChannel. The problem
>>>>>>> seems to lock contention reading from FlumeEventQueue:
>>>>>>>
>>>>>>> I see a lot of threads like this:
>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>>> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
>>>>>>> [0x00007f821afee000]
>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>         at
>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>>>>>>>         - waiting to lock <0x0000000518ee4c90> (a
>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>         at
>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>>>>>>>         at
>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>         at
>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>         at
>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>         at
>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>         at
>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>>
>>>>>>> I have two file channels and 8 Avro Sinks per file channel. I added
>>>>>>> more sinks because they weren't draining fast enough. It seems like they
>>>>>>> send the batch then wait for an ack before sending again, thus sends are
>>>>>>> not pipelined and having more sinks seemed like a good way of getting some
>>>>>>> parallelism.
>>>>>>>
>>>>>>> Here's the full stack trace:
>>>>>>> 2013-08-13 15:30:32
>>>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02 mixed
>>>>>>> mode):
>>>>>>>
>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
>>>>>>>    java.lang.Thread.State: WAITING (parking)
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> - parking to wait for  <0x000000056d932120> (a
>>>>>>> java.util.concurrent.CountDownLatch$Sync)
>>>>>>>  at
>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>>>>>>> at
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>>>>>>>  at
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>>>>>>> at
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>>>>>>>  at
>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
>>>>>>> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
>>>>>>>  at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
>>>>>>> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
>>>>>>>  at
>>>>>>> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
>>>>>>> at $Proxy7.appendBatch(Unknown Source)
>>>>>>>  at
>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
>>>>>>> at
>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
>>>>>>>  at
>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000 nid=0x41a1
>>>>>>> waiting on condition [0x0000000000000000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>
>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>> - parking to wait for  <0x00000005194c3340> (a
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>  at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>  at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> - parking to wait for  <0x00000005191a16d8> (a
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>  at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>  at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> - parking to wait for  <0x0000000519174e08> (a
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>  at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>  at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> - parking to wait for  <0x0000000519174e98> (a
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>  at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>  at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> - parking to wait for  <0x00000005195f9670> (a
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>  at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>  at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> - parking to wait for  <0x0000000519174ce8> (a
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>  at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>  at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> - parking to wait for  <0x00000005191a1918> (a
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>  at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>  at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> - parking to wait for  <0x00000005191e7f60> (a
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>  at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>  at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153
>>>>>>> runnable [0x00007f8210e4d000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>  - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
>>>>>>> - locked <0x000000054a2917a8> (a
>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>  - locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>> at
>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> - parking to wait for  <0x0000000519174f58> (a
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>  at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>  at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146
>>>>>>> runnable [0x00007f8210f4e000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>  - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
>>>>>>> - locked <0x0000000546e5dd28> (a
>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>  - locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>> at
>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118
>>>>>>> runnable [0x00007f821104f000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>  - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
>>>>>>> - locked <0x0000000544f763c0> (a
>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>  - locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>> at
>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111
>>>>>>> runnable [0x00007f8211150000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>  - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
>>>>>>> - locked <0x000000054473d5a8> (a
>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>  - locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>> at
>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110
>>>>>>> runnable [0x00007f8211251000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>  - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
>>>>>>> - locked <0x00000005437f4f70> (a
>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>  - locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>> at
>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f
>>>>>>> runnable [0x00007f8211352000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>  - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
>>>>>>> - locked <0x00000005422a9018> (a
>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>  - locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>> at
>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e
>>>>>>> runnable [0x00007f8211453000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>  - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
>>>>>>> - locked <0x0000000541063d50> (a
>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>  - locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>> at
>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d
>>>>>>> runnable [0x00007f8211554000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>  - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
>>>>>>> - locked <0x0000000540dc1110> (a
>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>  - locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>> at
>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a
>>>>>>> runnable [0x00007f8211655000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>  - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
>>>>>>> - locked <0x000000053fbbc3b0> (a
>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>  - locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>> at
>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109
>>>>>>> runnable [0x00007f8211756000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>  - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
>>>>>>> - locked <0x000000053f812860> (a
>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>  - locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>> at
>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd
>>>>>>> runnable [0x00007f8211857000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>  - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
>>>>>>> - locked <0x000000053e6e5570> (a
>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>  - locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>> at
>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> - parking to wait for  <0x0000000519175138> (a
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>  at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>  at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> - parking to wait for  <0x0000000519174c28> (a
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>  at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>  at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> - parking to wait for  <0x00000005191751c8> (a
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>  at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>  at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0
>>>>>>> runnable [0x00007f8211c5b000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>  - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
>>>>>>> - locked <0x0000000536579db0> (a
>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>  - locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>> at
>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> - parking to wait for  <0x00000005191750a8> (a
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>  at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>  at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1
>>>>>>> runnable [0x00007f8211e5d000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>  - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
>>>>>>> - locked <0x00000005313943a8> (a
>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>  - locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>> at
>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> - parking to wait for  <0x000000051936f460> (a
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>  at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>  at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> - parking to wait for  <0x000000051936dfa8> (a
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>  at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>  at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de
>>>>>>> runnable [0x00007f8212160000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>  - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
>>>>>>> - locked <0x000000052dad4468> (a
>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>  - locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>> at
>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd
>>>>>>> runnable [0x00007f8212261000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>  - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
>>>>>>> - locked <0x000000052c73c5f0> (a
>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>  - locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>> at
>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc
>>>>>>> runnable [0x00007f8212362000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>  - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
>>>>>>> - locked <0x000000052b8da998> (a
>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>  - locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>> at
>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db
>>>>>>> waiting for monitor entry [0x00007f8212462000]
>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>> at
>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>>>>>>> - waiting to lock <0x0000000518ee4c90> (a
>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>  at
>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>>>>>>> at
>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>>>>>>  at
>>>>>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>>>>>> at
>>>>>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>>>>>>  at
>>>>>>> org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>>>>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>>>>>>  at
>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>  at
>>>>>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>>>>>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>>>>>>  at
>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>  at
>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>>>>>>  at
>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>>>>>> at
>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>>>>>>  at
>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>  at
>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>> - parking to wait for  <0x000000051936df18> (a
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>  at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>  at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>> tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000]
>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> - parking to wait for  <0x0000000518fe7ae8> (a
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>  at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>  at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8
>>>>>>> runnable [0x00007f8212766000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>  - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
>>>>>>> - locked <0x00000005263d6ba8> (a
>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>  - locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>> at
>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7
>>>>>>> runnable [0x00007f8212867000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>  - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
>>>>>>> - locked <0x0000000525d1d0e0> (a
>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>  - locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>> at
>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6
>>>>>>> runnable [0x00007f8212968000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>  - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
>>>>>>> - locked <0x0000000525a689d8> (a
>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>  - locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>> at
>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5
>>>>>>> runnable [0x00007f8212a69000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>  - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
>>>>>>> - locked <0x0000000524e1e7d8> (a
>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>  - locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>> at
>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>  at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>
>>>>>>> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4
>>>>>>> runnable [0x00007f8212b6a000]
>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>  - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
>>>>>>> - locked <0x0000000524d437a8> (a
>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>  - locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>> at
>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>  at
>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>> at
>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>
>>>>>>> ...

Re: Lock contention in FileChannel

[Pankaj Gupta <pa...@brightroll.com>]

Both disks were at around 15-25%.


On Tue, Aug 13, 2013 at 7:54 PM, Brock Noland <br...@cloudera.com> wrote:

> Gotcha. When you run tge test what is tye disk utilization percentage?
> Iostat can be used for this.
> On Aug 13, 2013 9:47 PM, "Pankaj Gupta" <pa...@brightroll.com> wrote:
>
>> Those are the boxes we want to collect data from. They run flume and send
>> data through their avro sinks to the avro source on this box. We are
>> getting data at a pretty good rate and the problem is in fact that the
>> events don't drain from the FileChannel fast enough and the channel fill
>> percentage keeps getting higher.
>>
>>
>> On Tue, Aug 13, 2013 at 7:41 PM, Brock Noland <br...@cloudera.com> wrote:
>>
>>> What is sending the events to the avro source?
>>> On Aug 13, 2013 9:34 PM, "Pankaj Gupta" <pa...@brightroll.com> wrote:
>>>
>>>> Here's the config:
>>>> # define channels, one for each disk
>>>>
>>>>
>>>>
>>>>
>>>> agent1.channels.ch1.type = FILE
>>>> agent1.channels.ch1.checkpointDir = /flume1/checkpoint
>>>> agent1.channels.ch1.dataDirs = /flume1/data
>>>> agent1.channels.ch1.maxFileSize = 375809638400
>>>> agent1.channels.ch1.capacity = 75000000
>>>> agent1.channels.ch1.transactionCapacity = 4000
>>>>
>>>> agent1.channels.ch2.type = FILE
>>>> agent1.channels.ch2.checkpointDir = /flume2/checkpoint
>>>> agent1.channels.ch2.dataDirs = /flume2/data
>>>> agent1.channels.ch2.maxFileSize = 375809638400
>>>> agent1.channels.ch2.capacity = 75000000
>>>> agent1.channels.ch2.transactionCapacity = 4000
>>>>
>>>>
>>>>
>>>> # Define an Avro source named avroSource1
>>>> # Each sink can connect to only one channel.
>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>
>>>>
>>>> agent1.sources.avroSource1.channels = ch1
>>>> agent1.sources.avroSource1.type = avro
>>>> agent1.sources.avroSource1.bind = 0.0.0.0
>>>> agent1.sources.avroSource1.port = <port>
>>>>
>>>>
>>>>
>>>>
>>>> agent1.sinks.avroSink1-1-1.type = avro
>>>> agent1.sinks.avroSink1-1-1.channel = ch1
>>>> agent1.sinks.avroSink1-1-1.hostname = <hostname>
>>>> agent1.sinks.avroSink1-1-1.port = <port>
>>>> agent1.sinks.avroSink1-1-1.connect-timeout = 300000
>>>> agent1.sinks.avroSink1-1-1.batch-size = 4000
>>>>
>>>>
>>>>
>>>>
>>>> agent1.sinks.avroSink1-2-1.type = avro
>>>> agent1.sinks.avroSink1-2-1.channel = ch1
>>>> agent1.sinks.avroSink1-2-1.hostname = <hostname>
>>>> agent1.sinks.avroSink1-2-1.port = <port>
>>>> agent1.sinks.avroSink1-2-1.connect-timeout = 300000
>>>> agent1.sinks.avroSink1-2-1.batch-size = 4000
>>>>
>>>>
>>>>
>>>>
>>>> agent1.sinks.avroSink1-3-1.type = avro
>>>> agent1.sinks.avroSink1-3-1.channel = ch1
>>>> agent1.sinks.avroSink1-3-1.hostname = <hostname>
>>>> agent1.sinks.avroSink1-3-1.port = <port>
>>>> agent1.sinks.avroSink1-3-1.connect-timeout = 300000
>>>> agent1.sinks.avroSink1-3-1.batch-size = 4000
>>>>
>>>>
>>>>
>>>>
>>>> agent1.sinks.avroSink1-4-1.type = avro
>>>> agent1.sinks.avroSink1-4-1.channel = ch1
>>>> agent1.sinks.avroSink1-4-1.hostname = <hostname>
>>>> agent1.sinks.avroSink1-4-1.port = <port>
>>>> agent1.sinks.avroSink1-4-1.connect-timeout = 300000
>>>> agent1.sinks.avroSink1-4-1.batch-size = 4000
>>>>
>>>>
>>>>
>>>> #Add the sink groups; load-balance between each group of sinks which
>>>> round robin between different hops
>>>> agent1.sinkgroups.group1.sinks = avroSink1-1-1 avroSink1-2-1
>>>> avroSink1-3-1 avroSink1-4-1
>>>> agent1.sinkgroups.group1.processor.type = load_balance
>>>> agent1.sinkgroups.group1.processor.selector = ROUND_ROBIN
>>>> agent1.sinkgroups.group1.processor.backoff = true
>>>>
>>>>
>>>> #End of set
>>>>
>>>> # Define an Avro source named avroSource2
>>>> # Each sink can connect to only one channel.
>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>
>>>>
>>>> agent1.sources.avroSource2.channels = ch2
>>>> agent1.sources.avroSource2.type = avro
>>>> agent1.sources.avroSource2.bind = 0.0.0.0
>>>> agent1.sources.avroSource2.port = <port>
>>>>
>>>>
>>>>
>>>>
>>>> agent1.sinks.avroSink2-1-1.type = avro
>>>> agent1.sinks.avroSink2-1-1.channel = ch2
>>>> agent1.sinks.avroSink2-1-1.hostname = <hostname>
>>>> agent1.sinks.avroSink2-1-1.port = <port>
>>>> agent1.sinks.avroSink2-1-1.connect-timeout = 300000
>>>> agent1.sinks.avroSink2-1-1.batch-size = 4000
>>>>
>>>>
>>>>
>>>>
>>>> agent1.sinks.avroSink2-2-1.type = avro
>>>> agent1.sinks.avroSink2-2-1.channel = ch2
>>>> agent1.sinks.avroSink2-2-1.hostname = <hostname>
>>>> agent1.sinks.avroSink2-2-1.port = <port>
>>>> agent1.sinks.avroSink2-2-1.connect-timeout = 300000
>>>> agent1.sinks.avroSink2-2-1.batch-size = 4000
>>>>
>>>>
>>>>
>>>>
>>>> agent1.sinks.avroSink2-3-1.type = avro
>>>> agent1.sinks.avroSink2-3-1.channel = ch2
>>>> agent1.sinks.avroSink2-3-1.hostname = <hostname>
>>>> agent1.sinks.avroSink2-3-1.port = <port>
>>>> agent1.sinks.avroSink2-3-1.connect-timeout = 300000
>>>> agent1.sinks.avroSink2-3-1.batch-size = 4000
>>>>
>>>>
>>>>
>>>>
>>>> agent1.sinks.avroSink2-4-1.type = avro
>>>> agent1.sinks.avroSink2-4-1.channel = ch2
>>>> agent1.sinks.avroSink2-4-1.hostname = <hostname>
>>>> agent1.sinks.avroSink2-4-1.port = <port>
>>>> agent1.sinks.avroSink2-4-1.connect-timeout = 300000
>>>> agent1.sinks.avroSink2-4-1.batch-size = 4000
>>>>
>>>>
>>>>
>>>> #Add the sink groups; load-balance between each group of sinks which
>>>> round robin between different hops
>>>> agent1.sinkgroups.group2.sinks = avroSink2-1-1 avroSink2-2-1
>>>> avroSink2-3-1 avroSink2-4-1
>>>> agent1.sinkgroups.group2.processor.type = load_balance
>>>> agent1.sinkgroups.group2.processor.selector = ROUND_ROBIN
>>>> agent1.sinkgroups.group2.processor.backoff = true
>>>>
>>>>
>>>> #End of set
>>>>
>>>> # Define an Avro source named avroSource3
>>>> # Each sink can connect to only one channel.
>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>
>>>>
>>>> agent1.sources.avroSource3.channels = ch1
>>>> agent1.sources.avroSource3.type = avro
>>>> agent1.sources.avroSource3.bind = 0.0.0.0
>>>> agent1.sources.avroSource3.port = <port>
>>>>
>>>>
>>>>
>>>>
>>>> agent1.sinks.avroSink3-1-1.type = avro
>>>> agent1.sinks.avroSink3-1-1.channel = ch1
>>>> agent1.sinks.avroSink3-1-1.hostname = <hostname>
>>>> agent1.sinks.avroSink3-1-1.port = <port>
>>>> agent1.sinks.avroSink3-1-1.connect-timeout = 300000
>>>> agent1.sinks.avroSink3-1-1.batch-size = 4000
>>>>
>>>>
>>>>
>>>>
>>>> agent1.sinks.avroSink3-2-1.type = avro
>>>> agent1.sinks.avroSink3-2-1.channel = ch1
>>>> agent1.sinks.avroSink3-2-1.hostname = <hostname>
>>>> agent1.sinks.avroSink3-2-1.port = <port>
>>>> agent1.sinks.avroSink3-2-1.connect-timeout = 300000
>>>> agent1.sinks.avroSink3-2-1.batch-size = 4000
>>>>
>>>>
>>>>
>>>>
>>>> agent1.sinks.avroSink3-3-1.type = avro
>>>> agent1.sinks.avroSink3-3-1.channel = ch1
>>>> agent1.sinks.avroSink3-3-1.hostname = <hostname>
>>>> agent1.sinks.avroSink3-3-1.port = <port>
>>>> agent1.sinks.avroSink3-3-1.connect-timeout = 300000
>>>> agent1.sinks.avroSink3-3-1.batch-size = 4000
>>>>
>>>>
>>>>
>>>>
>>>> agent1.sinks.avroSink3-4-1.type = avro
>>>> agent1.sinks.avroSink3-4-1.channel = ch1
>>>> agent1.sinks.avroSink3-4-1.hostname = <hostname>
>>>> agent1.sinks.avroSink3-4-1.port = <port>
>>>> agent1.sinks.avroSink3-4-1.connect-timeout = 300000
>>>> agent1.sinks.avroSink3-4-1.batch-size = 4000
>>>>
>>>>
>>>>
>>>> #Add the sink groups; load-balance between each group of sinks which
>>>> round robin between different hops
>>>> agent1.sinkgroups.group3.sinks = avroSink3-1-1 avroSink3-2-1
>>>> avroSink3-3-1 avroSink3-4-1
>>>> agent1.sinkgroups.group3.processor.type = load_balance
>>>> agent1.sinkgroups.group3.processor.selector = ROUND_ROBIN
>>>> agent1.sinkgroups.group3.processor.backoff = true
>>>>
>>>>
>>>> #End of set
>>>>
>>>> # Define an Avro source named avroSource4
>>>> # Each sink can connect to only one channel.
>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>
>>>>
>>>> agent1.sources.avroSource4.channels = ch2
>>>> agent1.sources.avroSource4.type = avro
>>>> agent1.sources.avroSource4.bind = 0.0.0.0
>>>> agent1.sources.avroSource4.port = <port>
>>>>
>>>>
>>>>
>>>>
>>>> agent1.sinks.avroSink4-1-1.type = avro
>>>> agent1.sinks.avroSink4-1-1.channel = ch2
>>>> agent1.sinks.avroSink4-1-1.hostname = <hostname>
>>>> agent1.sinks.avroSink4-1-1.port = <port>
>>>> agent1.sinks.avroSink4-1-1.connect-timeout = 300000
>>>> agent1.sinks.avroSink4-1-1.batch-size = 4000
>>>>
>>>>
>>>>
>>>>
>>>> agent1.sinks.avroSink4-2-1.type = avro
>>>> agent1.sinks.avroSink4-2-1.channel = ch2
>>>> agent1.sinks.avroSink4-2-1.hostname = <hostname>
>>>> agent1.sinks.avroSink4-2-1.port = <port>
>>>> agent1.sinks.avroSink4-2-1.connect-timeout = 300000
>>>> agent1.sinks.avroSink4-2-1.batch-size = 4000
>>>>
>>>>
>>>>
>>>>
>>>> agent1.sinks.avroSink4-3-1.type = avro
>>>> agent1.sinks.avroSink4-3-1.channel = ch2
>>>> agent1.sinks.avroSink4-3-1.hostname = <hostname>
>>>> agent1.sinks.avroSink4-3-1.port = <port>
>>>> agent1.sinks.avroSink4-3-1.connect-timeout = 300000
>>>> agent1.sinks.avroSink4-3-1.batch-size = 4000
>>>>
>>>>
>>>>
>>>>
>>>> agent1.sinks.avroSink4-4-1.type = avro
>>>> agent1.sinks.avroSink4-4-1.channel = ch2
>>>> agent1.sinks.avroSink4-4-1.hostname = <hostname>
>>>> agent1.sinks.avroSink4-4-1.port = <port>
>>>> agent1.sinks.avroSink4-4-1.connect-timeout = 300000
>>>> agent1.sinks.avroSink4-4-1.batch-size = 4000
>>>>
>>>>
>>>>
>>>> #Add the sink groups; load-balance between each group of sinks which
>>>> round robin between different hops
>>>> agent1.sinkgroups.group4.sinks = avroSink4-1-1 avroSink4-2-1
>>>> avroSink4-3-1 avroSink4-4-1
>>>> agent1.sinkgroups.group4.processor.type = load_balance
>>>> agent1.sinkgroups.group4.processor.selector = ROUND_ROBIN
>>>> agent1.sinkgroups.group4.processor.backoff = true
>>>>
>>>>
>>>> #End of set
>>>>
>>>>
>>>>
>>>> # Finally, now that we've defined all of our components, tell
>>>> # agent1 which ones we want to activate.
>>>> agent1.channels = ch1 ch2
>>>> agent1.sources = avroSource1 avroSource2 avroSource3 avroSource4
>>>> agent1.sinks = avroSink1-1-1 avroSink1-2-1 avroSink1-3-1 avroSink1-4-1
>>>> avroSink2-1-1 avroSink2-2-1 avroSink2-3-1 avroSink2-4-1 avroSink3-1-1
>>>> avroSink3-2-1 avroSink3-3-1 avroSink3-4-1 avroSink4-1-1 avroSink4-2-1
>>>> avroSink4-3-1 avroSink4-4-1
>>>> agent1.sinkgroups = group1 group2 group3 group4
>>>>
>>>> There are two disks on the box /flume1 and /flume2
>>>> 32 cores of CPU, 32 GB RAM, 1GbPS ethernet.
>>>>
>>>>
>>>> On Tue, Aug 13, 2013 at 7:22 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>
>>>>> Can you share your conf file?
>>>>> On Aug 13, 2013 9:19 PM, "Hari Shreedharan" <hs...@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>>  Even though the writes are done per batch, they don't go to disk
>>>>>> rightaway - commits are the only ones which actually cause an fsync - which
>>>>>> is when writes actually go to disk.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Hari
>>>>>>
>>>>>> On Tuesday, August 13, 2013 at 7:06 PM, Pankaj Gupta wrote:
>>>>>>
>>>>>> Looking at the code it seems like the lock and the i/o is done per
>>>>>> event and not for a batch. Is that correct? If that is the case then it
>>>>>> seems like there is a lot of overhead per event. The throughput I'm seeing
>>>>>> is 1 - 1.5 MBps per disk which is way below the sequential read/write
>>>>>> capacity of the disk which is easily over 50MBps. Adding more sinks doesn't
>>>>>> help, they just block waiting for the queue to become free. CPU usage is
>>>>>> 20%, there is enough RAM for page cache so that no read is going to disk.
>>>>>> The queue seems to be the bottleneck. What is the throughput I should
>>>>>> expect per disk?
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>>
>>>>>> The lock is per file. Adding more directories to the channel will
>>>>>> cause more files to be created. Of course you'll need additional disks
>>>>>> behind those directories to see any performance increase.
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <
>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>
>>>>>>  Yes, IO is done inside locks to avoid multiple takes and puts
>>>>>> getting written out at the same time. Even though Java makes sure the
>>>>>> writes are serialized, Flume still needs to keep track of some counters
>>>>>> etc, so the lock is required. Note that the lock you are talking about is
>>>>>>  in the LogFile class, which represents a single file - so even if the
>>>>>> write is inside that lock (which is also inside that class itself) that
>>>>>>  does not cause any contention - because the lock is just preventing two IO
>>>>>> ops to happen at the same time.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Hari
>>>>>>
>>>>>> On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:
>>>>>>
>>>>>> It seems like some i/o is done inside the lock, which means that time
>>>>>> for taking a lock is proportional to the time for i/o and thus it becomes a
>>>>>> problem. I apologize in advance if I am wrong but the call stack and
>>>>>> behavior I'm seeing seems to suggest that. Specifically, it seems that we
>>>>>> do a write while inside take:
>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>         at sun.nio.ch.NativeThread.current(Native Method)
>>>>>>         at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>>>>>>         at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>>>>>>         - locked <0x00000005190ec998> (a java.lang.Object)
>>>>>>         at
>>>>>> org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>>>>>>         at
>>>>>> org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>>>>>>         - locked <0x0000000519111590> (a
>>>>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>>>>>         at org.apache.flume.channel.file.Log.take(Log.java:550)
>>>>>>         at
>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>>>>>>         at
>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>         at
>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>         at
>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>         at
>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>         at
>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <
>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>
>>>>>> Since the channel is designed to make sure that events are not
>>>>>> duplicated to multiple sinks, and to protect against corruption due to
>>>>>> concurrency issues, we do not need the locking in the channel's flume event
>>>>>> queue. It is unlikely that locking is what is causing performance issues
>>>>>> because the channel is heavily I/O bound. If you take a series of thread
>>>>>> dumps, you will probably see that those threads are moving forward and the
>>>>>> ones reading/writing from/to disk are the ones which are slower. These
>>>>>> locks are unlikely to hit performance much.
>>>>>>
>>>>>> Thanks,
>>>>>> Hari
>>>>>>
>>>>>> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Spent some more time debugging issues with FileChannel. The problem
>>>>>> seems to lock contention reading from FlumeEventQueue:
>>>>>>
>>>>>> I see a lot of threads like this:
>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
>>>>>> [0x00007f821afee000]
>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>         at
>>>>>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>>>>>>         - waiting to lock <0x0000000518ee4c90> (a
>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>         at
>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>>>>>>         at
>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>         at
>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>         at
>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>         at
>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>         at
>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>>
>>>>>> I have two file channels and 8 Avro Sinks per file channel. I added
>>>>>> more sinks because they weren't draining fast enough. It seems like they
>>>>>> send the batch then wait for an ack before sending again, thus sends are
>>>>>> not pipelined and having more sinks seemed like a good way of getting some
>>>>>> parallelism.
>>>>>>
>>>>>> Here's the full stack trace:
>>>>>> 2013-08-13 15:30:32
>>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02 mixed
>>>>>> mode):
>>>>>>
>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
>>>>>>    java.lang.Thread.State: WAITING (parking)
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> - parking to wait for  <0x000000056d932120> (a
>>>>>> java.util.concurrent.CountDownLatch$Sync)
>>>>>>  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>>>>>> at
>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>>>>>>  at
>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>>>>>> at
>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>>>>>>  at
>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
>>>>>> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
>>>>>>  at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
>>>>>> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
>>>>>>  at
>>>>>> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
>>>>>> at $Proxy7.appendBatch(Unknown Source)
>>>>>>  at
>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
>>>>>> at
>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
>>>>>>  at
>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000 nid=0x41a1
>>>>>> waiting on condition [0x0000000000000000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>
>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>> - parking to wait for  <0x00000005194c3340> (a
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>  at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>  at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> - parking to wait for  <0x00000005191a16d8> (a
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>  at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>  at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> - parking to wait for  <0x0000000519174e08> (a
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>  at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>  at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> - parking to wait for  <0x0000000519174e98> (a
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>  at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>  at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> - parking to wait for  <0x00000005195f9670> (a
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>  at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>  at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> - parking to wait for  <0x0000000519174ce8> (a
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>  at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>  at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> - parking to wait for  <0x00000005191a1918> (a
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>  at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>  at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> - parking to wait for  <0x00000005191e7f60> (a
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>  at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>  at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153
>>>>>> runnable [0x00007f8210e4d000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>  - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
>>>>>> - locked <0x000000054a2917a8> (a
>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>  - locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl)
>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>> at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> - parking to wait for  <0x0000000519174f58> (a
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>  at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>  at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146
>>>>>> runnable [0x00007f8210f4e000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>  - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
>>>>>> - locked <0x0000000546e5dd28> (a
>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>  - locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>> at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118
>>>>>> runnable [0x00007f821104f000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>  - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
>>>>>> - locked <0x0000000544f763c0> (a
>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>  - locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>> at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111
>>>>>> runnable [0x00007f8211150000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>  - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
>>>>>> - locked <0x000000054473d5a8> (a
>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>  - locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl)
>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>> at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110
>>>>>> runnable [0x00007f8211251000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>  - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
>>>>>> - locked <0x00000005437f4f70> (a
>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>  - locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl)
>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>> at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f
>>>>>> runnable [0x00007f8211352000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>  - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
>>>>>> - locked <0x00000005422a9018> (a
>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>  - locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl)
>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>> at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e
>>>>>> runnable [0x00007f8211453000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>  - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
>>>>>> - locked <0x0000000541063d50> (a
>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>  - locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl)
>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>> at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d
>>>>>> runnable [0x00007f8211554000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>  - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
>>>>>> - locked <0x0000000540dc1110> (a
>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>  - locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl)
>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>> at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a
>>>>>> runnable [0x00007f8211655000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>  - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
>>>>>> - locked <0x000000053fbbc3b0> (a
>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>  - locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl)
>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>> at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109
>>>>>> runnable [0x00007f8211756000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>  - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
>>>>>> - locked <0x000000053f812860> (a
>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>  - locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>> at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd
>>>>>> runnable [0x00007f8211857000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>  - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
>>>>>> - locked <0x000000053e6e5570> (a
>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>  - locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl)
>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>> at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> - parking to wait for  <0x0000000519175138> (a
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>  at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>  at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> - parking to wait for  <0x0000000519174c28> (a
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>  at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>  at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> - parking to wait for  <0x00000005191751c8> (a
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>  at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>  at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0
>>>>>> runnable [0x00007f8211c5b000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>  - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
>>>>>> - locked <0x0000000536579db0> (a
>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>  - locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl)
>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>> at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> - parking to wait for  <0x00000005191750a8> (a
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>  at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>  at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1
>>>>>> runnable [0x00007f8211e5d000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>  - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
>>>>>> - locked <0x00000005313943a8> (a
>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>  - locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl)
>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>> at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> - parking to wait for  <0x000000051936f460> (a
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>  at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>  at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> - parking to wait for  <0x000000051936dfa8> (a
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>  at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>  at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de
>>>>>> runnable [0x00007f8212160000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>  - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
>>>>>> - locked <0x000000052dad4468> (a
>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>  - locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl)
>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>> at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd
>>>>>> runnable [0x00007f8212261000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>  - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
>>>>>> - locked <0x000000052c73c5f0> (a
>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>  - locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl)
>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>> at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc
>>>>>> runnable [0x00007f8212362000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>  - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
>>>>>> - locked <0x000000052b8da998> (a
>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>  - locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl)
>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>> at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db
>>>>>> waiting for monitor entry [0x00007f8212462000]
>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>> at
>>>>>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>>>>>> - waiting to lock <0x0000000518ee4c90> (a
>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>  at
>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>>>>>> at
>>>>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>>>>>  at
>>>>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>>>>> at
>>>>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>>>>>  at
>>>>>> org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>>>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>>>>>  at
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>  at
>>>>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>>>>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>>>>>  at
>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>>>>> at
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>  at
>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>>>>> at
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>  at
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>>>>> at
>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>>>>>  at
>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>>>>> at
>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>>>>>  at
>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>>>>> at
>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>  at
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>> at
>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>>>>>  at
>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>>>>> at
>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>  at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>> - parking to wait for  <0x000000051936df18> (a
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>  at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>  at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>> tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000]
>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> - parking to wait for  <0x0000000518fe7ae8> (a
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>  at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>  at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8
>>>>>> runnable [0x00007f8212766000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>  - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
>>>>>> - locked <0x00000005263d6ba8> (a
>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>  - locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl)
>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>> at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7
>>>>>> runnable [0x00007f8212867000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>  - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
>>>>>> - locked <0x0000000525d1d0e0> (a
>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>  - locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>> at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6
>>>>>> runnable [0x00007f8212968000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>  - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
>>>>>> - locked <0x0000000525a689d8> (a
>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>  - locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl)
>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>> at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5
>>>>>> runnable [0x00007f8212a69000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>  - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
>>>>>> - locked <0x0000000524e1e7d8> (a
>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>  - locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl)
>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>> at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4
>>>>>> runnable [0x00007f8212b6a000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>  - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
>>>>>> - locked <0x0000000524d437a8> (a
>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>  - locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl)
>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>> at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>  at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> "pool-142-thread-36" prio=10 tid=0x00007f85843ad800 nid=0x40d3
>>>>>> runnable [0x00007f8212c6b000]
>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>  - locked <0x0000000524cf4140> (a sun.nio.ch.Util$2)
>>>>>> - locked <0x0000000524cf4128> (a
>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>  - locked <0x0000000524ab32e8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>> at
>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>  at
>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>> at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>
>>>>>> ...
>
>


-- 


*P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com

Pankaj Gupta | Software Engineer

*BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com


United States | Canada | United Kingdom | Germany


We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
!

Re: Lock contention in FileChannel

[Brock Noland <br...@cloudera.com>]

Gotcha. When you run tge test what is tye disk utilization percentage?
Iostat can be used for this.
On Aug 13, 2013 9:47 PM, "Pankaj Gupta" <pa...@brightroll.com> wrote:

> Those are the boxes we want to collect data from. They run flume and send
> data through their avro sinks to the avro source on this box. We are
> getting data at a pretty good rate and the problem is in fact that the
> events don't drain from the FileChannel fast enough and the channel fill
> percentage keeps getting higher.
>
>
> On Tue, Aug 13, 2013 at 7:41 PM, Brock Noland <br...@cloudera.com> wrote:
>
>> What is sending the events to the avro source?
>> On Aug 13, 2013 9:34 PM, "Pankaj Gupta" <pa...@brightroll.com> wrote:
>>
>>> Here's the config:
>>> # define channels, one for each disk
>>>
>>>
>>>
>>>
>>> agent1.channels.ch1.type = FILE
>>> agent1.channels.ch1.checkpointDir = /flume1/checkpoint
>>> agent1.channels.ch1.dataDirs = /flume1/data
>>> agent1.channels.ch1.maxFileSize = 375809638400
>>> agent1.channels.ch1.capacity = 75000000
>>> agent1.channels.ch1.transactionCapacity = 4000
>>>
>>> agent1.channels.ch2.type = FILE
>>> agent1.channels.ch2.checkpointDir = /flume2/checkpoint
>>> agent1.channels.ch2.dataDirs = /flume2/data
>>> agent1.channels.ch2.maxFileSize = 375809638400
>>> agent1.channels.ch2.capacity = 75000000
>>> agent1.channels.ch2.transactionCapacity = 4000
>>>
>>>
>>>
>>> # Define an Avro source named avroSource1
>>> # Each sink can connect to only one channel.
>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>
>>>
>>> agent1.sources.avroSource1.channels = ch1
>>> agent1.sources.avroSource1.type = avro
>>> agent1.sources.avroSource1.bind = 0.0.0.0
>>> agent1.sources.avroSource1.port = <port>
>>>
>>>
>>>
>>>
>>> agent1.sinks.avroSink1-1-1.type = avro
>>> agent1.sinks.avroSink1-1-1.channel = ch1
>>> agent1.sinks.avroSink1-1-1.hostname = <hostname>
>>> agent1.sinks.avroSink1-1-1.port = <port>
>>> agent1.sinks.avroSink1-1-1.connect-timeout = 300000
>>> agent1.sinks.avroSink1-1-1.batch-size = 4000
>>>
>>>
>>>
>>>
>>> agent1.sinks.avroSink1-2-1.type = avro
>>> agent1.sinks.avroSink1-2-1.channel = ch1
>>> agent1.sinks.avroSink1-2-1.hostname = <hostname>
>>> agent1.sinks.avroSink1-2-1.port = <port>
>>> agent1.sinks.avroSink1-2-1.connect-timeout = 300000
>>> agent1.sinks.avroSink1-2-1.batch-size = 4000
>>>
>>>
>>>
>>>
>>> agent1.sinks.avroSink1-3-1.type = avro
>>> agent1.sinks.avroSink1-3-1.channel = ch1
>>> agent1.sinks.avroSink1-3-1.hostname = <hostname>
>>> agent1.sinks.avroSink1-3-1.port = <port>
>>> agent1.sinks.avroSink1-3-1.connect-timeout = 300000
>>> agent1.sinks.avroSink1-3-1.batch-size = 4000
>>>
>>>
>>>
>>>
>>> agent1.sinks.avroSink1-4-1.type = avro
>>> agent1.sinks.avroSink1-4-1.channel = ch1
>>> agent1.sinks.avroSink1-4-1.hostname = <hostname>
>>> agent1.sinks.avroSink1-4-1.port = <port>
>>> agent1.sinks.avroSink1-4-1.connect-timeout = 300000
>>> agent1.sinks.avroSink1-4-1.batch-size = 4000
>>>
>>>
>>>
>>> #Add the sink groups; load-balance between each group of sinks which
>>> round robin between different hops
>>> agent1.sinkgroups.group1.sinks = avroSink1-1-1 avroSink1-2-1
>>> avroSink1-3-1 avroSink1-4-1
>>> agent1.sinkgroups.group1.processor.type = load_balance
>>> agent1.sinkgroups.group1.processor.selector = ROUND_ROBIN
>>> agent1.sinkgroups.group1.processor.backoff = true
>>>
>>>
>>> #End of set
>>>
>>> # Define an Avro source named avroSource2
>>> # Each sink can connect to only one channel.
>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>
>>>
>>> agent1.sources.avroSource2.channels = ch2
>>> agent1.sources.avroSource2.type = avro
>>> agent1.sources.avroSource2.bind = 0.0.0.0
>>> agent1.sources.avroSource2.port = <port>
>>>
>>>
>>>
>>>
>>> agent1.sinks.avroSink2-1-1.type = avro
>>> agent1.sinks.avroSink2-1-1.channel = ch2
>>> agent1.sinks.avroSink2-1-1.hostname = <hostname>
>>> agent1.sinks.avroSink2-1-1.port = <port>
>>> agent1.sinks.avroSink2-1-1.connect-timeout = 300000
>>> agent1.sinks.avroSink2-1-1.batch-size = 4000
>>>
>>>
>>>
>>>
>>> agent1.sinks.avroSink2-2-1.type = avro
>>> agent1.sinks.avroSink2-2-1.channel = ch2
>>> agent1.sinks.avroSink2-2-1.hostname = <hostname>
>>> agent1.sinks.avroSink2-2-1.port = <port>
>>> agent1.sinks.avroSink2-2-1.connect-timeout = 300000
>>> agent1.sinks.avroSink2-2-1.batch-size = 4000
>>>
>>>
>>>
>>>
>>> agent1.sinks.avroSink2-3-1.type = avro
>>> agent1.sinks.avroSink2-3-1.channel = ch2
>>> agent1.sinks.avroSink2-3-1.hostname = <hostname>
>>> agent1.sinks.avroSink2-3-1.port = <port>
>>> agent1.sinks.avroSink2-3-1.connect-timeout = 300000
>>> agent1.sinks.avroSink2-3-1.batch-size = 4000
>>>
>>>
>>>
>>>
>>> agent1.sinks.avroSink2-4-1.type = avro
>>> agent1.sinks.avroSink2-4-1.channel = ch2
>>> agent1.sinks.avroSink2-4-1.hostname = <hostname>
>>> agent1.sinks.avroSink2-4-1.port = <port>
>>> agent1.sinks.avroSink2-4-1.connect-timeout = 300000
>>> agent1.sinks.avroSink2-4-1.batch-size = 4000
>>>
>>>
>>>
>>> #Add the sink groups; load-balance between each group of sinks which
>>> round robin between different hops
>>> agent1.sinkgroups.group2.sinks = avroSink2-1-1 avroSink2-2-1
>>> avroSink2-3-1 avroSink2-4-1
>>> agent1.sinkgroups.group2.processor.type = load_balance
>>> agent1.sinkgroups.group2.processor.selector = ROUND_ROBIN
>>> agent1.sinkgroups.group2.processor.backoff = true
>>>
>>>
>>> #End of set
>>>
>>> # Define an Avro source named avroSource3
>>> # Each sink can connect to only one channel.
>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>
>>>
>>> agent1.sources.avroSource3.channels = ch1
>>> agent1.sources.avroSource3.type = avro
>>> agent1.sources.avroSource3.bind = 0.0.0.0
>>> agent1.sources.avroSource3.port = <port>
>>>
>>>
>>>
>>>
>>> agent1.sinks.avroSink3-1-1.type = avro
>>> agent1.sinks.avroSink3-1-1.channel = ch1
>>> agent1.sinks.avroSink3-1-1.hostname = <hostname>
>>> agent1.sinks.avroSink3-1-1.port = <port>
>>> agent1.sinks.avroSink3-1-1.connect-timeout = 300000
>>> agent1.sinks.avroSink3-1-1.batch-size = 4000
>>>
>>>
>>>
>>>
>>> agent1.sinks.avroSink3-2-1.type = avro
>>> agent1.sinks.avroSink3-2-1.channel = ch1
>>> agent1.sinks.avroSink3-2-1.hostname = <hostname>
>>> agent1.sinks.avroSink3-2-1.port = <port>
>>> agent1.sinks.avroSink3-2-1.connect-timeout = 300000
>>> agent1.sinks.avroSink3-2-1.batch-size = 4000
>>>
>>>
>>>
>>>
>>> agent1.sinks.avroSink3-3-1.type = avro
>>> agent1.sinks.avroSink3-3-1.channel = ch1
>>> agent1.sinks.avroSink3-3-1.hostname = <hostname>
>>> agent1.sinks.avroSink3-3-1.port = <port>
>>> agent1.sinks.avroSink3-3-1.connect-timeout = 300000
>>> agent1.sinks.avroSink3-3-1.batch-size = 4000
>>>
>>>
>>>
>>>
>>> agent1.sinks.avroSink3-4-1.type = avro
>>> agent1.sinks.avroSink3-4-1.channel = ch1
>>> agent1.sinks.avroSink3-4-1.hostname = <hostname>
>>> agent1.sinks.avroSink3-4-1.port = <port>
>>> agent1.sinks.avroSink3-4-1.connect-timeout = 300000
>>> agent1.sinks.avroSink3-4-1.batch-size = 4000
>>>
>>>
>>>
>>> #Add the sink groups; load-balance between each group of sinks which
>>> round robin between different hops
>>> agent1.sinkgroups.group3.sinks = avroSink3-1-1 avroSink3-2-1
>>> avroSink3-3-1 avroSink3-4-1
>>> agent1.sinkgroups.group3.processor.type = load_balance
>>> agent1.sinkgroups.group3.processor.selector = ROUND_ROBIN
>>> agent1.sinkgroups.group3.processor.backoff = true
>>>
>>>
>>> #End of set
>>>
>>> # Define an Avro source named avroSource4
>>> # Each sink can connect to only one channel.
>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>
>>>
>>> agent1.sources.avroSource4.channels = ch2
>>> agent1.sources.avroSource4.type = avro
>>> agent1.sources.avroSource4.bind = 0.0.0.0
>>> agent1.sources.avroSource4.port = <port>
>>>
>>>
>>>
>>>
>>> agent1.sinks.avroSink4-1-1.type = avro
>>> agent1.sinks.avroSink4-1-1.channel = ch2
>>> agent1.sinks.avroSink4-1-1.hostname = <hostname>
>>> agent1.sinks.avroSink4-1-1.port = <port>
>>> agent1.sinks.avroSink4-1-1.connect-timeout = 300000
>>> agent1.sinks.avroSink4-1-1.batch-size = 4000
>>>
>>>
>>>
>>>
>>> agent1.sinks.avroSink4-2-1.type = avro
>>> agent1.sinks.avroSink4-2-1.channel = ch2
>>> agent1.sinks.avroSink4-2-1.hostname = <hostname>
>>> agent1.sinks.avroSink4-2-1.port = <port>
>>> agent1.sinks.avroSink4-2-1.connect-timeout = 300000
>>> agent1.sinks.avroSink4-2-1.batch-size = 4000
>>>
>>>
>>>
>>>
>>> agent1.sinks.avroSink4-3-1.type = avro
>>> agent1.sinks.avroSink4-3-1.channel = ch2
>>> agent1.sinks.avroSink4-3-1.hostname = <hostname>
>>> agent1.sinks.avroSink4-3-1.port = <port>
>>> agent1.sinks.avroSink4-3-1.connect-timeout = 300000
>>> agent1.sinks.avroSink4-3-1.batch-size = 4000
>>>
>>>
>>>
>>>
>>> agent1.sinks.avroSink4-4-1.type = avro
>>> agent1.sinks.avroSink4-4-1.channel = ch2
>>> agent1.sinks.avroSink4-4-1.hostname = <hostname>
>>> agent1.sinks.avroSink4-4-1.port = <port>
>>> agent1.sinks.avroSink4-4-1.connect-timeout = 300000
>>> agent1.sinks.avroSink4-4-1.batch-size = 4000
>>>
>>>
>>>
>>> #Add the sink groups; load-balance between each group of sinks which
>>> round robin between different hops
>>> agent1.sinkgroups.group4.sinks = avroSink4-1-1 avroSink4-2-1
>>> avroSink4-3-1 avroSink4-4-1
>>> agent1.sinkgroups.group4.processor.type = load_balance
>>> agent1.sinkgroups.group4.processor.selector = ROUND_ROBIN
>>> agent1.sinkgroups.group4.processor.backoff = true
>>>
>>>
>>> #End of set
>>>
>>>
>>>
>>> # Finally, now that we've defined all of our components, tell
>>> # agent1 which ones we want to activate.
>>> agent1.channels = ch1 ch2
>>> agent1.sources = avroSource1 avroSource2 avroSource3 avroSource4
>>> agent1.sinks = avroSink1-1-1 avroSink1-2-1 avroSink1-3-1 avroSink1-4-1
>>> avroSink2-1-1 avroSink2-2-1 avroSink2-3-1 avroSink2-4-1 avroSink3-1-1
>>> avroSink3-2-1 avroSink3-3-1 avroSink3-4-1 avroSink4-1-1 avroSink4-2-1
>>> avroSink4-3-1 avroSink4-4-1
>>> agent1.sinkgroups = group1 group2 group3 group4
>>>
>>> There are two disks on the box /flume1 and /flume2
>>> 32 cores of CPU, 32 GB RAM, 1GbPS ethernet.
>>>
>>>
>>> On Tue, Aug 13, 2013 at 7:22 PM, Brock Noland <br...@cloudera.com>wrote:
>>>
>>>> Can you share your conf file?
>>>> On Aug 13, 2013 9:19 PM, "Hari Shreedharan" <hs...@cloudera.com>
>>>> wrote:
>>>>
>>>>>  Even though the writes are done per batch, they don't go to disk
>>>>> rightaway - commits are the only ones which actually cause an fsync - which
>>>>> is when writes actually go to disk.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Hari
>>>>>
>>>>> On Tuesday, August 13, 2013 at 7:06 PM, Pankaj Gupta wrote:
>>>>>
>>>>> Looking at the code it seems like the lock and the i/o is done per
>>>>> event and not for a batch. Is that correct? If that is the case then it
>>>>> seems like there is a lot of overhead per event. The throughput I'm seeing
>>>>> is 1 - 1.5 MBps per disk which is way below the sequential read/write
>>>>> capacity of the disk which is easily over 50MBps. Adding more sinks doesn't
>>>>> help, they just block waiting for the queue to become free. CPU usage is
>>>>> 20%, there is enough RAM for page cache so that no read is going to disk.
>>>>> The queue seems to be the bottleneck. What is the throughput I should
>>>>> expect per disk?
>>>>>
>>>>>
>>>>> On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>>
>>>>> The lock is per file. Adding more directories to the channel will
>>>>> cause more files to be created. Of course you'll need additional disks
>>>>> behind those directories to see any performance increase.
>>>>>
>>>>>
>>>>> On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <
>>>>> hshreedharan@cloudera.com> wrote:
>>>>>
>>>>>  Yes, IO is done inside locks to avoid multiple takes and puts getting
>>>>> written out at the same time. Even though Java makes sure the writes are
>>>>> serialized, Flume still needs to keep track of some counters etc, so the
>>>>> lock is required. Note that the lock you are talking about is  in the
>>>>> LogFile class, which represents a single file - so even if the write is
>>>>> inside that lock (which is also inside that class itself) that  does not
>>>>> cause any contention - because the lock is just preventing two IO ops to
>>>>> happen at the same time.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Hari
>>>>>
>>>>> On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:
>>>>>
>>>>> It seems like some i/o is done inside the lock, which means that time
>>>>> for taking a lock is proportional to the time for i/o and thus it becomes a
>>>>> problem. I apologize in advance if I am wrong but the call stack and
>>>>> behavior I'm seeing seems to suggest that. Specifically, it seems that we
>>>>> do a write while inside take:
>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>         at sun.nio.ch.NativeThread.current(Native Method)
>>>>>         at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>>>>>         at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>>>>>         - locked <0x00000005190ec998> (a java.lang.Object)
>>>>>         at
>>>>> org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>>>>>         at
>>>>> org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>>>>>         - locked <0x0000000519111590> (a
>>>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>>>>         at org.apache.flume.channel.file.Log.take(Log.java:550)
>>>>>         at
>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>>>>>         at
>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>         at
>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>         at
>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>         at
>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>         at
>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <
>>>>> hshreedharan@cloudera.com> wrote:
>>>>>
>>>>> Since the channel is designed to make sure that events are not
>>>>> duplicated to multiple sinks, and to protect against corruption due to
>>>>> concurrency issues, we do not need the locking in the channel's flume event
>>>>> queue. It is unlikely that locking is what is causing performance issues
>>>>> because the channel is heavily I/O bound. If you take a series of thread
>>>>> dumps, you will probably see that those threads are moving forward and the
>>>>> ones reading/writing from/to disk are the ones which are slower. These
>>>>> locks are unlikely to hit performance much.
>>>>>
>>>>> Thanks,
>>>>> Hari
>>>>>
>>>>> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> Spent some more time debugging issues with FileChannel. The problem
>>>>> seems to lock contention reading from FlumeEventQueue:
>>>>>
>>>>> I see a lot of threads like this:
>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
>>>>> [0x00007f821afee000]
>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>         at
>>>>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>>>>>         - waiting to lock <0x0000000518ee4c90> (a
>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>         at
>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>>>>>         at
>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>         at
>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>         at
>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>         at
>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>         at
>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>>
>>>>> I have two file channels and 8 Avro Sinks per file channel. I added
>>>>> more sinks because they weren't draining fast enough. It seems like they
>>>>> send the batch then wait for an ack before sending again, thus sends are
>>>>> not pipelined and having more sinks seemed like a good way of getting some
>>>>> parallelism.
>>>>>
>>>>> Here's the full stack trace:
>>>>> 2013-08-13 15:30:32
>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02 mixed
>>>>> mode):
>>>>>
>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
>>>>>    java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x000000056d932120> (a
>>>>> java.util.concurrent.CountDownLatch$Sync)
>>>>>  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>>>>> at
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>>>>>  at
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>>>>> at
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>>>>>  at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
>>>>> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
>>>>>  at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
>>>>> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
>>>>>  at
>>>>> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
>>>>> at $Proxy7.appendBatch(Unknown Source)
>>>>>  at
>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
>>>>> at
>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
>>>>>  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000 nid=0x41a1
>>>>> waiting on condition [0x0000000000000000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>
>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000005194c3340> (a
>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>  at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>  at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000005191a16d8> (a
>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>  at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>  at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000519174e08> (a
>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>  at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>  at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000519174e98> (a
>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>  at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>  at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000005195f9670> (a
>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>  at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>  at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000519174ce8> (a
>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>  at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>  at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000005191a1918> (a
>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>  at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>  at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000005191e7f60> (a
>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>  at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>  at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153
>>>>> runnable [0x00007f8210e4d000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>  - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000054a2917a8> (a java.util.Collections$UnmodifiableSet)
>>>>>  - locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000519174f58> (a
>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>  at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>  at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146
>>>>> runnable [0x00007f8210f4e000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>  - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
>>>>> - locked <0x0000000546e5dd28> (a java.util.Collections$UnmodifiableSet)
>>>>>  - locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118
>>>>> runnable [0x00007f821104f000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>  - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
>>>>> - locked <0x0000000544f763c0> (a java.util.Collections$UnmodifiableSet)
>>>>>  - locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111
>>>>> runnable [0x00007f8211150000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>  - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000054473d5a8> (a java.util.Collections$UnmodifiableSet)
>>>>>  - locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110
>>>>> runnable [0x00007f8211251000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>  - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000005437f4f70> (a java.util.Collections$UnmodifiableSet)
>>>>>  - locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f
>>>>> runnable [0x00007f8211352000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>  - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000005422a9018> (a java.util.Collections$UnmodifiableSet)
>>>>>  - locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e
>>>>> runnable [0x00007f8211453000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>  - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
>>>>> - locked <0x0000000541063d50> (a java.util.Collections$UnmodifiableSet)
>>>>>  - locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d
>>>>> runnable [0x00007f8211554000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>  - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
>>>>> - locked <0x0000000540dc1110> (a java.util.Collections$UnmodifiableSet)
>>>>>  - locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a
>>>>> runnable [0x00007f8211655000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>  - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000053fbbc3b0> (a java.util.Collections$UnmodifiableSet)
>>>>>  - locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109
>>>>> runnable [0x00007f8211756000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>  - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000053f812860> (a java.util.Collections$UnmodifiableSet)
>>>>>  - locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd
>>>>> runnable [0x00007f8211857000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>  - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000053e6e5570> (a java.util.Collections$UnmodifiableSet)
>>>>>  - locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000519175138> (a
>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>  at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>  at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000519174c28> (a
>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>  at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>  at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000005191751c8> (a
>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>  at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>  at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0
>>>>> runnable [0x00007f8211c5b000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>  - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
>>>>> - locked <0x0000000536579db0> (a java.util.Collections$UnmodifiableSet)
>>>>>  - locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x00000005191750a8> (a
>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>  at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>  at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1
>>>>> runnable [0x00007f8211e5d000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>  - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000005313943a8> (a java.util.Collections$UnmodifiableSet)
>>>>>  - locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x000000051936f460> (a
>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>  at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>  at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x000000051936dfa8> (a
>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>  at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>  at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de
>>>>> runnable [0x00007f8212160000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>  - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000052dad4468> (a java.util.Collections$UnmodifiableSet)
>>>>>  - locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd
>>>>> runnable [0x00007f8212261000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>  - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000052c73c5f0> (a java.util.Collections$UnmodifiableSet)
>>>>>  - locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc
>>>>> runnable [0x00007f8212362000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>  - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
>>>>> - locked <0x000000052b8da998> (a java.util.Collections$UnmodifiableSet)
>>>>>  - locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db waiting
>>>>> for monitor entry [0x00007f8212462000]
>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>> at
>>>>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>>>>> - waiting to lock <0x0000000518ee4c90> (a
>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>  at
>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>>>>> at
>>>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>>>>  at
>>>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>>>> at
>>>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>>>>  at
>>>>> org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>>>>  at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>  at
>>>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>>>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>>>>  at
>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>>>> at
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>  at
>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>>>> at
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>  at
>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>>>> at
>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>>>>  at
>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>>>> at
>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>>>>  at
>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>>>> at
>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>  at
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>> at
>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>>>>  at
>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>>>> at
>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>  at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x000000051936df18> (a
>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>  at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>  at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>> tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000]
>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for  <0x0000000518fe7ae8> (a
>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>  at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>  at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8
>>>>> runnable [0x00007f8212766000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>  - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
>>>>> - locked <0x00000005263d6ba8> (a java.util.Collections$UnmodifiableSet)
>>>>>  - locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7
>>>>> runnable [0x00007f8212867000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>  - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
>>>>> - locked <0x0000000525d1d0e0> (a java.util.Collections$UnmodifiableSet)
>>>>>  - locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6
>>>>> runnable [0x00007f8212968000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>  - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
>>>>> - locked <0x0000000525a689d8> (a java.util.Collections$UnmodifiableSet)
>>>>>  - locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5
>>>>> runnable [0x00007f8212a69000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>  - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
>>>>> - locked <0x0000000524e1e7d8> (a java.util.Collections$UnmodifiableSet)
>>>>>  - locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4
>>>>> runnable [0x00007f8212b6a000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>  - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
>>>>> - locked <0x0000000524d437a8> (a java.util.Collections$UnmodifiableSet)
>>>>>  - locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>  at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> "pool-142-thread-36" prio=10 tid=0x00007f85843ad800 nid=0x40d3
>>>>> runnable [0x00007f8212c6b000]
>>>>>    java.lang.Thread.State: RUNNABLE
>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>  - locked <0x0000000524cf4140> (a sun.nio.ch.Util$2)
>>>>> - locked <0x0000000524cf4128> (a java.util.Collections$UnmodifiableSet)
>>>>>  - locked <0x0000000524ab32e8> (a sun.nio.ch.EPollSelectorImpl)
>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>  at
>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>
>>>>> ...

Re: Lock contention in FileChannel

[Pankaj Gupta <pa...@brightroll.com>]

Those are the boxes we want to collect data from. They run flume and send
data through their avro sinks to the avro source on this box. We are
getting data at a pretty good rate and the problem is in fact that the
events don't drain from the FileChannel fast enough and the channel fill
percentage keeps getting higher.


On Tue, Aug 13, 2013 at 7:41 PM, Brock Noland <br...@cloudera.com> wrote:

> What is sending the events to the avro source?
> On Aug 13, 2013 9:34 PM, "Pankaj Gupta" <pa...@brightroll.com> wrote:
>
>> Here's the config:
>> # define channels, one for each disk
>>
>>
>>
>>
>> agent1.channels.ch1.type = FILE
>> agent1.channels.ch1.checkpointDir = /flume1/checkpoint
>> agent1.channels.ch1.dataDirs = /flume1/data
>> agent1.channels.ch1.maxFileSize = 375809638400
>> agent1.channels.ch1.capacity = 75000000
>> agent1.channels.ch1.transactionCapacity = 4000
>>
>> agent1.channels.ch2.type = FILE
>> agent1.channels.ch2.checkpointDir = /flume2/checkpoint
>> agent1.channels.ch2.dataDirs = /flume2/data
>> agent1.channels.ch2.maxFileSize = 375809638400
>> agent1.channels.ch2.capacity = 75000000
>> agent1.channels.ch2.transactionCapacity = 4000
>>
>>
>>
>> # Define an Avro source named avroSource1
>> # Each sink can connect to only one channel.
>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>
>>
>> agent1.sources.avroSource1.channels = ch1
>> agent1.sources.avroSource1.type = avro
>> agent1.sources.avroSource1.bind = 0.0.0.0
>> agent1.sources.avroSource1.port = <port>
>>
>>
>>
>>
>> agent1.sinks.avroSink1-1-1.type = avro
>> agent1.sinks.avroSink1-1-1.channel = ch1
>> agent1.sinks.avroSink1-1-1.hostname = <hostname>
>> agent1.sinks.avroSink1-1-1.port = <port>
>> agent1.sinks.avroSink1-1-1.connect-timeout = 300000
>> agent1.sinks.avroSink1-1-1.batch-size = 4000
>>
>>
>>
>>
>> agent1.sinks.avroSink1-2-1.type = avro
>> agent1.sinks.avroSink1-2-1.channel = ch1
>> agent1.sinks.avroSink1-2-1.hostname = <hostname>
>> agent1.sinks.avroSink1-2-1.port = <port>
>> agent1.sinks.avroSink1-2-1.connect-timeout = 300000
>> agent1.sinks.avroSink1-2-1.batch-size = 4000
>>
>>
>>
>>
>> agent1.sinks.avroSink1-3-1.type = avro
>> agent1.sinks.avroSink1-3-1.channel = ch1
>> agent1.sinks.avroSink1-3-1.hostname = <hostname>
>> agent1.sinks.avroSink1-3-1.port = <port>
>> agent1.sinks.avroSink1-3-1.connect-timeout = 300000
>> agent1.sinks.avroSink1-3-1.batch-size = 4000
>>
>>
>>
>>
>> agent1.sinks.avroSink1-4-1.type = avro
>> agent1.sinks.avroSink1-4-1.channel = ch1
>> agent1.sinks.avroSink1-4-1.hostname = <hostname>
>> agent1.sinks.avroSink1-4-1.port = <port>
>> agent1.sinks.avroSink1-4-1.connect-timeout = 300000
>> agent1.sinks.avroSink1-4-1.batch-size = 4000
>>
>>
>>
>> #Add the sink groups; load-balance between each group of sinks which
>> round robin between different hops
>> agent1.sinkgroups.group1.sinks = avroSink1-1-1 avroSink1-2-1
>> avroSink1-3-1 avroSink1-4-1
>> agent1.sinkgroups.group1.processor.type = load_balance
>> agent1.sinkgroups.group1.processor.selector = ROUND_ROBIN
>> agent1.sinkgroups.group1.processor.backoff = true
>>
>>
>> #End of set
>>
>> # Define an Avro source named avroSource2
>> # Each sink can connect to only one channel.
>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>
>>
>> agent1.sources.avroSource2.channels = ch2
>> agent1.sources.avroSource2.type = avro
>> agent1.sources.avroSource2.bind = 0.0.0.0
>> agent1.sources.avroSource2.port = <port>
>>
>>
>>
>>
>> agent1.sinks.avroSink2-1-1.type = avro
>> agent1.sinks.avroSink2-1-1.channel = ch2
>> agent1.sinks.avroSink2-1-1.hostname = <hostname>
>> agent1.sinks.avroSink2-1-1.port = <port>
>> agent1.sinks.avroSink2-1-1.connect-timeout = 300000
>> agent1.sinks.avroSink2-1-1.batch-size = 4000
>>
>>
>>
>>
>> agent1.sinks.avroSink2-2-1.type = avro
>> agent1.sinks.avroSink2-2-1.channel = ch2
>> agent1.sinks.avroSink2-2-1.hostname = <hostname>
>> agent1.sinks.avroSink2-2-1.port = <port>
>> agent1.sinks.avroSink2-2-1.connect-timeout = 300000
>> agent1.sinks.avroSink2-2-1.batch-size = 4000
>>
>>
>>
>>
>> agent1.sinks.avroSink2-3-1.type = avro
>> agent1.sinks.avroSink2-3-1.channel = ch2
>> agent1.sinks.avroSink2-3-1.hostname = <hostname>
>> agent1.sinks.avroSink2-3-1.port = <port>
>> agent1.sinks.avroSink2-3-1.connect-timeout = 300000
>> agent1.sinks.avroSink2-3-1.batch-size = 4000
>>
>>
>>
>>
>> agent1.sinks.avroSink2-4-1.type = avro
>> agent1.sinks.avroSink2-4-1.channel = ch2
>> agent1.sinks.avroSink2-4-1.hostname = <hostname>
>> agent1.sinks.avroSink2-4-1.port = <port>
>> agent1.sinks.avroSink2-4-1.connect-timeout = 300000
>> agent1.sinks.avroSink2-4-1.batch-size = 4000
>>
>>
>>
>> #Add the sink groups; load-balance between each group of sinks which
>> round robin between different hops
>> agent1.sinkgroups.group2.sinks = avroSink2-1-1 avroSink2-2-1
>> avroSink2-3-1 avroSink2-4-1
>> agent1.sinkgroups.group2.processor.type = load_balance
>> agent1.sinkgroups.group2.processor.selector = ROUND_ROBIN
>> agent1.sinkgroups.group2.processor.backoff = true
>>
>>
>> #End of set
>>
>> # Define an Avro source named avroSource3
>> # Each sink can connect to only one channel.
>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>
>>
>> agent1.sources.avroSource3.channels = ch1
>> agent1.sources.avroSource3.type = avro
>> agent1.sources.avroSource3.bind = 0.0.0.0
>> agent1.sources.avroSource3.port = <port>
>>
>>
>>
>>
>> agent1.sinks.avroSink3-1-1.type = avro
>> agent1.sinks.avroSink3-1-1.channel = ch1
>> agent1.sinks.avroSink3-1-1.hostname = <hostname>
>> agent1.sinks.avroSink3-1-1.port = <port>
>> agent1.sinks.avroSink3-1-1.connect-timeout = 300000
>> agent1.sinks.avroSink3-1-1.batch-size = 4000
>>
>>
>>
>>
>> agent1.sinks.avroSink3-2-1.type = avro
>> agent1.sinks.avroSink3-2-1.channel = ch1
>> agent1.sinks.avroSink3-2-1.hostname = <hostname>
>> agent1.sinks.avroSink3-2-1.port = <port>
>> agent1.sinks.avroSink3-2-1.connect-timeout = 300000
>> agent1.sinks.avroSink3-2-1.batch-size = 4000
>>
>>
>>
>>
>> agent1.sinks.avroSink3-3-1.type = avro
>> agent1.sinks.avroSink3-3-1.channel = ch1
>> agent1.sinks.avroSink3-3-1.hostname = <hostname>
>> agent1.sinks.avroSink3-3-1.port = <port>
>> agent1.sinks.avroSink3-3-1.connect-timeout = 300000
>> agent1.sinks.avroSink3-3-1.batch-size = 4000
>>
>>
>>
>>
>> agent1.sinks.avroSink3-4-1.type = avro
>> agent1.sinks.avroSink3-4-1.channel = ch1
>> agent1.sinks.avroSink3-4-1.hostname = <hostname>
>> agent1.sinks.avroSink3-4-1.port = <port>
>> agent1.sinks.avroSink3-4-1.connect-timeout = 300000
>> agent1.sinks.avroSink3-4-1.batch-size = 4000
>>
>>
>>
>> #Add the sink groups; load-balance between each group of sinks which
>> round robin between different hops
>> agent1.sinkgroups.group3.sinks = avroSink3-1-1 avroSink3-2-1
>> avroSink3-3-1 avroSink3-4-1
>> agent1.sinkgroups.group3.processor.type = load_balance
>> agent1.sinkgroups.group3.processor.selector = ROUND_ROBIN
>> agent1.sinkgroups.group3.processor.backoff = true
>>
>>
>> #End of set
>>
>> # Define an Avro source named avroSource4
>> # Each sink can connect to only one channel.
>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>
>>
>> agent1.sources.avroSource4.channels = ch2
>> agent1.sources.avroSource4.type = avro
>> agent1.sources.avroSource4.bind = 0.0.0.0
>> agent1.sources.avroSource4.port = <port>
>>
>>
>>
>>
>> agent1.sinks.avroSink4-1-1.type = avro
>> agent1.sinks.avroSink4-1-1.channel = ch2
>> agent1.sinks.avroSink4-1-1.hostname = <hostname>
>> agent1.sinks.avroSink4-1-1.port = <port>
>> agent1.sinks.avroSink4-1-1.connect-timeout = 300000
>> agent1.sinks.avroSink4-1-1.batch-size = 4000
>>
>>
>>
>>
>> agent1.sinks.avroSink4-2-1.type = avro
>> agent1.sinks.avroSink4-2-1.channel = ch2
>> agent1.sinks.avroSink4-2-1.hostname = <hostname>
>> agent1.sinks.avroSink4-2-1.port = <port>
>> agent1.sinks.avroSink4-2-1.connect-timeout = 300000
>> agent1.sinks.avroSink4-2-1.batch-size = 4000
>>
>>
>>
>>
>> agent1.sinks.avroSink4-3-1.type = avro
>> agent1.sinks.avroSink4-3-1.channel = ch2
>> agent1.sinks.avroSink4-3-1.hostname = <hostname>
>> agent1.sinks.avroSink4-3-1.port = <port>
>> agent1.sinks.avroSink4-3-1.connect-timeout = 300000
>> agent1.sinks.avroSink4-3-1.batch-size = 4000
>>
>>
>>
>>
>> agent1.sinks.avroSink4-4-1.type = avro
>> agent1.sinks.avroSink4-4-1.channel = ch2
>> agent1.sinks.avroSink4-4-1.hostname = <hostname>
>> agent1.sinks.avroSink4-4-1.port = <port>
>> agent1.sinks.avroSink4-4-1.connect-timeout = 300000
>> agent1.sinks.avroSink4-4-1.batch-size = 4000
>>
>>
>>
>> #Add the sink groups; load-balance between each group of sinks which
>> round robin between different hops
>> agent1.sinkgroups.group4.sinks = avroSink4-1-1 avroSink4-2-1
>> avroSink4-3-1 avroSink4-4-1
>> agent1.sinkgroups.group4.processor.type = load_balance
>> agent1.sinkgroups.group4.processor.selector = ROUND_ROBIN
>> agent1.sinkgroups.group4.processor.backoff = true
>>
>>
>> #End of set
>>
>>
>>
>> # Finally, now that we've defined all of our components, tell
>> # agent1 which ones we want to activate.
>> agent1.channels = ch1 ch2
>> agent1.sources = avroSource1 avroSource2 avroSource3 avroSource4
>> agent1.sinks = avroSink1-1-1 avroSink1-2-1 avroSink1-3-1 avroSink1-4-1
>> avroSink2-1-1 avroSink2-2-1 avroSink2-3-1 avroSink2-4-1 avroSink3-1-1
>> avroSink3-2-1 avroSink3-3-1 avroSink3-4-1 avroSink4-1-1 avroSink4-2-1
>> avroSink4-3-1 avroSink4-4-1
>> agent1.sinkgroups = group1 group2 group3 group4
>>
>> There are two disks on the box /flume1 and /flume2
>> 32 cores of CPU, 32 GB RAM, 1GbPS ethernet.
>>
>>
>> On Tue, Aug 13, 2013 at 7:22 PM, Brock Noland <br...@cloudera.com> wrote:
>>
>>> Can you share your conf file?
>>> On Aug 13, 2013 9:19 PM, "Hari Shreedharan" <hs...@cloudera.com>
>>> wrote:
>>>
>>>>  Even though the writes are done per batch, they don't go to disk
>>>> rightaway - commits are the only ones which actually cause an fsync - which
>>>> is when writes actually go to disk.
>>>>
>>>>
>>>> Thanks,
>>>> Hari
>>>>
>>>> On Tuesday, August 13, 2013 at 7:06 PM, Pankaj Gupta wrote:
>>>>
>>>> Looking at the code it seems like the lock and the i/o is done per
>>>> event and not for a batch. Is that correct? If that is the case then it
>>>> seems like there is a lot of overhead per event. The throughput I'm seeing
>>>> is 1 - 1.5 MBps per disk which is way below the sequential read/write
>>>> capacity of the disk which is easily over 50MBps. Adding more sinks doesn't
>>>> help, they just block waiting for the queue to become free. CPU usage is
>>>> 20%, there is enough RAM for page cache so that no read is going to disk.
>>>> The queue seems to be the bottleneck. What is the throughput I should
>>>> expect per disk?
>>>>
>>>>
>>>> On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland <br...@cloudera.com>wrote:
>>>>
>>>> The lock is per file. Adding more directories to the channel will cause
>>>> more files to be created. Of course you'll need additional disks behind
>>>> those directories to see any performance increase.
>>>>
>>>>
>>>> On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <
>>>> hshreedharan@cloudera.com> wrote:
>>>>
>>>>  Yes, IO is done inside locks to avoid multiple takes and puts getting
>>>> written out at the same time. Even though Java makes sure the writes are
>>>> serialized, Flume still needs to keep track of some counters etc, so the
>>>> lock is required. Note that the lock you are talking about is  in the
>>>> LogFile class, which represents a single file - so even if the write is
>>>> inside that lock (which is also inside that class itself) that  does not
>>>> cause any contention - because the lock is just preventing two IO ops to
>>>> happen at the same time.
>>>>
>>>>
>>>> Thanks,
>>>> Hari
>>>>
>>>> On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:
>>>>
>>>> It seems like some i/o is done inside the lock, which means that time
>>>> for taking a lock is proportional to the time for i/o and thus it becomes a
>>>> problem. I apologize in advance if I am wrong but the call stack and
>>>> behavior I'm seeing seems to suggest that. Specifically, it seems that we
>>>> do a write while inside take:
>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>>         at sun.nio.ch.NativeThread.current(Native Method)
>>>>         at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>>>>         at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>>>>         - locked <0x00000005190ec998> (a java.lang.Object)
>>>>         at
>>>> org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>>>>         at
>>>> org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>>>>         - locked <0x0000000519111590> (a
>>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>>>         at org.apache.flume.channel.file.Log.take(Log.java:550)
>>>>         at
>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>>>>         at
>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>         at
>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>         at
>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>         at
>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>         at
>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>
>>>>
>>>>
>>>> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <
>>>> hshreedharan@cloudera.com> wrote:
>>>>
>>>> Since the channel is designed to make sure that events are not
>>>> duplicated to multiple sinks, and to protect against corruption due to
>>>> concurrency issues, we do not need the locking in the channel's flume event
>>>> queue. It is unlikely that locking is what is causing performance issues
>>>> because the channel is heavily I/O bound. If you take a series of thread
>>>> dumps, you will probably see that those threads are moving forward and the
>>>> ones reading/writing from/to disk are the ones which are slower. These
>>>> locks are unlikely to hit performance much.
>>>>
>>>> Thanks,
>>>> Hari
>>>>
>>>> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
>>>>
>>>> Hi,
>>>>
>>>> Spent some more time debugging issues with FileChannel. The problem
>>>> seems to lock contention reading from FlumeEventQueue:
>>>>
>>>> I see a lot of threads like this:
>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
>>>> [0x00007f821afee000]
>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>         at
>>>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>>>>         - waiting to lock <0x0000000518ee4c90> (a
>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>         at
>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>>>>         at
>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>         at
>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>         at
>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>         at
>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>         at
>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>
>>>>
>>>> I have two file channels and 8 Avro Sinks per file channel. I added
>>>> more sinks because they weren't draining fast enough. It seems like they
>>>> send the batch then wait for an ack before sending again, thus sends are
>>>> not pipelined and having more sinks seemed like a good way of getting some
>>>> parallelism.
>>>>
>>>> Here's the full stack trace:
>>>> 2013-08-13 15:30:32
>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02 mixed
>>>> mode):
>>>>
>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
>>>>    java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x000000056d932120> (a
>>>> java.util.concurrent.CountDownLatch$Sync)
>>>>  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>>>> at
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>>>>  at
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>>>> at
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>>>>  at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
>>>> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
>>>>  at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
>>>> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
>>>>  at
>>>> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
>>>> at $Proxy7.appendBatch(Unknown Source)
>>>>  at
>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
>>>> at
>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
>>>>  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000 nid=0x41a1
>>>> waiting on condition [0x0000000000000000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>>
>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>  at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000005194c3340> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>  at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>  at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000005191a16d8> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>  at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>  at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x0000000519174e08> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>  at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>  at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x0000000519174e98> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>  at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>  at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000005195f9670> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>  at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>  at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x0000000519174ce8> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>  at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>  at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000005191a1918> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>  at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>  at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000005191e7f60> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>  at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>  at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153 runnable
>>>> [0x00007f8210e4d000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
>>>> - locked <0x000000054a2917a8> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x0000000519174f58> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>  at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>  at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146 runnable
>>>> [0x00007f8210f4e000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
>>>> - locked <0x0000000546e5dd28> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118 runnable
>>>> [0x00007f821104f000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
>>>> - locked <0x0000000544f763c0> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111 runnable
>>>> [0x00007f8211150000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
>>>> - locked <0x000000054473d5a8> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110 runnable
>>>> [0x00007f8211251000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
>>>> - locked <0x00000005437f4f70> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f runnable
>>>> [0x00007f8211352000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
>>>> - locked <0x00000005422a9018> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e runnable
>>>> [0x00007f8211453000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
>>>> - locked <0x0000000541063d50> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d runnable
>>>> [0x00007f8211554000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
>>>> - locked <0x0000000540dc1110> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a runnable
>>>> [0x00007f8211655000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
>>>> - locked <0x000000053fbbc3b0> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109 runnable
>>>> [0x00007f8211756000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
>>>> - locked <0x000000053f812860> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd runnable
>>>> [0x00007f8211857000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
>>>> - locked <0x000000053e6e5570> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x0000000519175138> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>  at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>  at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x0000000519174c28> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>  at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>  at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000005191751c8> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>  at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>  at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0 runnable
>>>> [0x00007f8211c5b000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
>>>> - locked <0x0000000536579db0> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x00000005191750a8> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>  at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>  at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1 runnable
>>>> [0x00007f8211e5d000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
>>>> - locked <0x00000005313943a8> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x000000051936f460> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>  at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>  at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x000000051936dfa8> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>  at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>  at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de runnable
>>>> [0x00007f8212160000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
>>>> - locked <0x000000052dad4468> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd runnable
>>>> [0x00007f8212261000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
>>>> - locked <0x000000052c73c5f0> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc runnable
>>>> [0x00007f8212362000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
>>>> - locked <0x000000052b8da998> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db waiting
>>>> for monitor entry [0x00007f8212462000]
>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>> at
>>>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>>>> - waiting to lock <0x0000000518ee4c90> (a
>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>  at
>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>>>> at
>>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>>>  at
>>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>>> at
>>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>>>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>>>  at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>  at
>>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>>>  at
>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>>> at
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>  at
>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>>> at
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>  at
>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>>> at
>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>>>  at
>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>>> at
>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>>>  at
>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>>> at
>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>  at
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>> at
>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>>>  at
>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>>> at
>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>  at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>  at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x000000051936df18> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>  at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>  at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>> tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000]
>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for  <0x0000000518fe7ae8> (a
>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>  at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>  at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8 runnable
>>>> [0x00007f8212766000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
>>>> - locked <0x00000005263d6ba8> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7 runnable
>>>> [0x00007f8212867000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
>>>> - locked <0x0000000525d1d0e0> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6 runnable
>>>> [0x00007f8212968000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
>>>> - locked <0x0000000525a689d8> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5 runnable
>>>> [0x00007f8212a69000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
>>>> - locked <0x0000000524e1e7d8> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4 runnable
>>>> [0x00007f8212b6a000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
>>>> - locked <0x0000000524d437a8> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-142-thread-36" prio=10 tid=0x00007f85843ad800 nid=0x40d3 runnable
>>>> [0x00007f8212c6b000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x0000000524cf4140> (a sun.nio.ch.Util$2)
>>>> - locked <0x0000000524cf4128> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x0000000524ab32e8> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> "pool-144-thread-35" prio=10 tid=0x00007f85d83a9000 nid=0x40d2 runnable
>>>> [0x00007f8212d6c000]
>>>>    java.lang.Thread.State: RUNNABLE
>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>  - locked <0x00000005244877c8> (a sun.nio.ch.Util$2)
>>>> - locked <0x00000005244877b0> (a java.util.Collections$UnmodifiableSet)
>>>>  - locked <0x0000000523f5b660> (a sun.nio.ch.EPollSelectorImpl)
>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>  at
>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>  at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>
>>>> ...
>
>


-- 


*P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com

Pankaj Gupta | Software Engineer

*BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com


United States | Canada | United Kingdom | Germany


We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
!

Re: Lock contention in FileChannel

[Brock Noland <br...@cloudera.com>]

What is sending the events to the avro source?
On Aug 13, 2013 9:34 PM, "Pankaj Gupta" <pa...@brightroll.com> wrote:

> Here's the config:
> # define channels, one for each disk
>
>
>
>
> agent1.channels.ch1.type = FILE
> agent1.channels.ch1.checkpointDir = /flume1/checkpoint
> agent1.channels.ch1.dataDirs = /flume1/data
> agent1.channels.ch1.maxFileSize = 375809638400
> agent1.channels.ch1.capacity = 75000000
> agent1.channels.ch1.transactionCapacity = 4000
>
> agent1.channels.ch2.type = FILE
> agent1.channels.ch2.checkpointDir = /flume2/checkpoint
> agent1.channels.ch2.dataDirs = /flume2/data
> agent1.channels.ch2.maxFileSize = 375809638400
> agent1.channels.ch2.capacity = 75000000
> agent1.channels.ch2.transactionCapacity = 4000
>
>
>
> # Define an Avro source named avroSource1
> # Each sink can connect to only one channel.
> # Connect it to channel ch1. Load balance it to 2 avroSinks
>
>
> agent1.sources.avroSource1.channels = ch1
> agent1.sources.avroSource1.type = avro
> agent1.sources.avroSource1.bind = 0.0.0.0
> agent1.sources.avroSource1.port = <port>
>
>
>
>
> agent1.sinks.avroSink1-1-1.type = avro
> agent1.sinks.avroSink1-1-1.channel = ch1
> agent1.sinks.avroSink1-1-1.hostname = <hostname>
> agent1.sinks.avroSink1-1-1.port = <port>
> agent1.sinks.avroSink1-1-1.connect-timeout = 300000
> agent1.sinks.avroSink1-1-1.batch-size = 4000
>
>
>
>
> agent1.sinks.avroSink1-2-1.type = avro
> agent1.sinks.avroSink1-2-1.channel = ch1
> agent1.sinks.avroSink1-2-1.hostname = <hostname>
> agent1.sinks.avroSink1-2-1.port = <port>
> agent1.sinks.avroSink1-2-1.connect-timeout = 300000
> agent1.sinks.avroSink1-2-1.batch-size = 4000
>
>
>
>
> agent1.sinks.avroSink1-3-1.type = avro
> agent1.sinks.avroSink1-3-1.channel = ch1
> agent1.sinks.avroSink1-3-1.hostname = <hostname>
> agent1.sinks.avroSink1-3-1.port = <port>
> agent1.sinks.avroSink1-3-1.connect-timeout = 300000
> agent1.sinks.avroSink1-3-1.batch-size = 4000
>
>
>
>
> agent1.sinks.avroSink1-4-1.type = avro
> agent1.sinks.avroSink1-4-1.channel = ch1
> agent1.sinks.avroSink1-4-1.hostname = <hostname>
> agent1.sinks.avroSink1-4-1.port = <port>
> agent1.sinks.avroSink1-4-1.connect-timeout = 300000
> agent1.sinks.avroSink1-4-1.batch-size = 4000
>
>
>
> #Add the sink groups; load-balance between each group of sinks which round
> robin between different hops
> agent1.sinkgroups.group1.sinks = avroSink1-1-1 avroSink1-2-1 avroSink1-3-1
> avroSink1-4-1
> agent1.sinkgroups.group1.processor.type = load_balance
> agent1.sinkgroups.group1.processor.selector = ROUND_ROBIN
> agent1.sinkgroups.group1.processor.backoff = true
>
>
> #End of set
>
> # Define an Avro source named avroSource2
> # Each sink can connect to only one channel.
> # Connect it to channel ch2. Load balance it to 2 avroSinks
>
>
> agent1.sources.avroSource2.channels = ch2
> agent1.sources.avroSource2.type = avro
> agent1.sources.avroSource2.bind = 0.0.0.0
> agent1.sources.avroSource2.port = <port>
>
>
>
>
> agent1.sinks.avroSink2-1-1.type = avro
> agent1.sinks.avroSink2-1-1.channel = ch2
> agent1.sinks.avroSink2-1-1.hostname = <hostname>
> agent1.sinks.avroSink2-1-1.port = <port>
> agent1.sinks.avroSink2-1-1.connect-timeout = 300000
> agent1.sinks.avroSink2-1-1.batch-size = 4000
>
>
>
>
> agent1.sinks.avroSink2-2-1.type = avro
> agent1.sinks.avroSink2-2-1.channel = ch2
> agent1.sinks.avroSink2-2-1.hostname = <hostname>
> agent1.sinks.avroSink2-2-1.port = <port>
> agent1.sinks.avroSink2-2-1.connect-timeout = 300000
> agent1.sinks.avroSink2-2-1.batch-size = 4000
>
>
>
>
> agent1.sinks.avroSink2-3-1.type = avro
> agent1.sinks.avroSink2-3-1.channel = ch2
> agent1.sinks.avroSink2-3-1.hostname = <hostname>
> agent1.sinks.avroSink2-3-1.port = <port>
> agent1.sinks.avroSink2-3-1.connect-timeout = 300000
> agent1.sinks.avroSink2-3-1.batch-size = 4000
>
>
>
>
> agent1.sinks.avroSink2-4-1.type = avro
> agent1.sinks.avroSink2-4-1.channel = ch2
> agent1.sinks.avroSink2-4-1.hostname = <hostname>
> agent1.sinks.avroSink2-4-1.port = <port>
> agent1.sinks.avroSink2-4-1.connect-timeout = 300000
> agent1.sinks.avroSink2-4-1.batch-size = 4000
>
>
>
> #Add the sink groups; load-balance between each group of sinks which round
> robin between different hops
> agent1.sinkgroups.group2.sinks = avroSink2-1-1 avroSink2-2-1 avroSink2-3-1
> avroSink2-4-1
> agent1.sinkgroups.group2.processor.type = load_balance
> agent1.sinkgroups.group2.processor.selector = ROUND_ROBIN
> agent1.sinkgroups.group2.processor.backoff = true
>
>
> #End of set
>
> # Define an Avro source named avroSource3
> # Each sink can connect to only one channel.
> # Connect it to channel ch1. Load balance it to 2 avroSinks
>
>
> agent1.sources.avroSource3.channels = ch1
> agent1.sources.avroSource3.type = avro
> agent1.sources.avroSource3.bind = 0.0.0.0
> agent1.sources.avroSource3.port = <port>
>
>
>
>
> agent1.sinks.avroSink3-1-1.type = avro
> agent1.sinks.avroSink3-1-1.channel = ch1
> agent1.sinks.avroSink3-1-1.hostname = <hostname>
> agent1.sinks.avroSink3-1-1.port = <port>
> agent1.sinks.avroSink3-1-1.connect-timeout = 300000
> agent1.sinks.avroSink3-1-1.batch-size = 4000
>
>
>
>
> agent1.sinks.avroSink3-2-1.type = avro
> agent1.sinks.avroSink3-2-1.channel = ch1
> agent1.sinks.avroSink3-2-1.hostname = <hostname>
> agent1.sinks.avroSink3-2-1.port = <port>
> agent1.sinks.avroSink3-2-1.connect-timeout = 300000
> agent1.sinks.avroSink3-2-1.batch-size = 4000
>
>
>
>
> agent1.sinks.avroSink3-3-1.type = avro
> agent1.sinks.avroSink3-3-1.channel = ch1
> agent1.sinks.avroSink3-3-1.hostname = <hostname>
> agent1.sinks.avroSink3-3-1.port = <port>
> agent1.sinks.avroSink3-3-1.connect-timeout = 300000
> agent1.sinks.avroSink3-3-1.batch-size = 4000
>
>
>
>
> agent1.sinks.avroSink3-4-1.type = avro
> agent1.sinks.avroSink3-4-1.channel = ch1
> agent1.sinks.avroSink3-4-1.hostname = <hostname>
> agent1.sinks.avroSink3-4-1.port = <port>
> agent1.sinks.avroSink3-4-1.connect-timeout = 300000
> agent1.sinks.avroSink3-4-1.batch-size = 4000
>
>
>
> #Add the sink groups; load-balance between each group of sinks which round
> robin between different hops
> agent1.sinkgroups.group3.sinks = avroSink3-1-1 avroSink3-2-1 avroSink3-3-1
> avroSink3-4-1
> agent1.sinkgroups.group3.processor.type = load_balance
> agent1.sinkgroups.group3.processor.selector = ROUND_ROBIN
> agent1.sinkgroups.group3.processor.backoff = true
>
>
> #End of set
>
> # Define an Avro source named avroSource4
> # Each sink can connect to only one channel.
> # Connect it to channel ch2. Load balance it to 2 avroSinks
>
>
> agent1.sources.avroSource4.channels = ch2
> agent1.sources.avroSource4.type = avro
> agent1.sources.avroSource4.bind = 0.0.0.0
> agent1.sources.avroSource4.port = <port>
>
>
>
>
> agent1.sinks.avroSink4-1-1.type = avro
> agent1.sinks.avroSink4-1-1.channel = ch2
> agent1.sinks.avroSink4-1-1.hostname = <hostname>
> agent1.sinks.avroSink4-1-1.port = <port>
> agent1.sinks.avroSink4-1-1.connect-timeout = 300000
> agent1.sinks.avroSink4-1-1.batch-size = 4000
>
>
>
>
> agent1.sinks.avroSink4-2-1.type = avro
> agent1.sinks.avroSink4-2-1.channel = ch2
> agent1.sinks.avroSink4-2-1.hostname = <hostname>
> agent1.sinks.avroSink4-2-1.port = <port>
> agent1.sinks.avroSink4-2-1.connect-timeout = 300000
> agent1.sinks.avroSink4-2-1.batch-size = 4000
>
>
>
>
> agent1.sinks.avroSink4-3-1.type = avro
> agent1.sinks.avroSink4-3-1.channel = ch2
> agent1.sinks.avroSink4-3-1.hostname = <hostname>
> agent1.sinks.avroSink4-3-1.port = <port>
> agent1.sinks.avroSink4-3-1.connect-timeout = 300000
> agent1.sinks.avroSink4-3-1.batch-size = 4000
>
>
>
>
> agent1.sinks.avroSink4-4-1.type = avro
> agent1.sinks.avroSink4-4-1.channel = ch2
> agent1.sinks.avroSink4-4-1.hostname = <hostname>
> agent1.sinks.avroSink4-4-1.port = <port>
> agent1.sinks.avroSink4-4-1.connect-timeout = 300000
> agent1.sinks.avroSink4-4-1.batch-size = 4000
>
>
>
> #Add the sink groups; load-balance between each group of sinks which round
> robin between different hops
> agent1.sinkgroups.group4.sinks = avroSink4-1-1 avroSink4-2-1 avroSink4-3-1
> avroSink4-4-1
> agent1.sinkgroups.group4.processor.type = load_balance
> agent1.sinkgroups.group4.processor.selector = ROUND_ROBIN
> agent1.sinkgroups.group4.processor.backoff = true
>
>
> #End of set
>
>
>
> # Finally, now that we've defined all of our components, tell
> # agent1 which ones we want to activate.
> agent1.channels = ch1 ch2
> agent1.sources = avroSource1 avroSource2 avroSource3 avroSource4
> agent1.sinks = avroSink1-1-1 avroSink1-2-1 avroSink1-3-1 avroSink1-4-1
> avroSink2-1-1 avroSink2-2-1 avroSink2-3-1 avroSink2-4-1 avroSink3-1-1
> avroSink3-2-1 avroSink3-3-1 avroSink3-4-1 avroSink4-1-1 avroSink4-2-1
> avroSink4-3-1 avroSink4-4-1
> agent1.sinkgroups = group1 group2 group3 group4
>
> There are two disks on the box /flume1 and /flume2
> 32 cores of CPU, 32 GB RAM, 1GbPS ethernet.
>
>
> On Tue, Aug 13, 2013 at 7:22 PM, Brock Noland <br...@cloudera.com> wrote:
>
>> Can you share your conf file?
>> On Aug 13, 2013 9:19 PM, "Hari Shreedharan" <hs...@cloudera.com>
>> wrote:
>>
>>>  Even though the writes are done per batch, they don't go to disk
>>> rightaway - commits are the only ones which actually cause an fsync - which
>>> is when writes actually go to disk.
>>>
>>>
>>> Thanks,
>>> Hari
>>>
>>> On Tuesday, August 13, 2013 at 7:06 PM, Pankaj Gupta wrote:
>>>
>>> Looking at the code it seems like the lock and the i/o is done per event
>>> and not for a batch. Is that correct? If that is the case then it seems
>>> like there is a lot of overhead per event. The throughput I'm seeing is 1 -
>>> 1.5 MBps per disk which is way below the sequential read/write capacity of
>>> the disk which is easily over 50MBps. Adding more sinks doesn't help, they
>>> just block waiting for the queue to become free. CPU usage is 20%, there is
>>> enough RAM for page cache so that no read is going to disk. The queue seems
>>> to be the bottleneck. What is the throughput I should expect per disk?
>>>
>>>
>>> On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland <br...@cloudera.com>wrote:
>>>
>>> The lock is per file. Adding more directories to the channel will cause
>>> more files to be created. Of course you'll need additional disks behind
>>> those directories to see any performance increase.
>>>
>>>
>>> On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <
>>> hshreedharan@cloudera.com> wrote:
>>>
>>>  Yes, IO is done inside locks to avoid multiple takes and puts getting
>>> written out at the same time. Even though Java makes sure the writes are
>>> serialized, Flume still needs to keep track of some counters etc, so the
>>> lock is required. Note that the lock you are talking about is  in the
>>> LogFile class, which represents a single file - so even if the write is
>>> inside that lock (which is also inside that class itself) that  does not
>>> cause any contention - because the lock is just preventing two IO ops to
>>> happen at the same time.
>>>
>>>
>>> Thanks,
>>> Hari
>>>
>>> On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:
>>>
>>> It seems like some i/o is done inside the lock, which means that time
>>> for taking a lock is proportional to the time for i/o and thus it becomes a
>>> problem. I apologize in advance if I am wrong but the call stack and
>>> behavior I'm seeing seems to suggest that. Specifically, it seems that we
>>> do a write while inside take:
>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>>>    java.lang.Thread.State: RUNNABLE
>>>         at sun.nio.ch.NativeThread.current(Native Method)
>>>         at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>>>         at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>>>         - locked <0x00000005190ec998> (a java.lang.Object)
>>>         at
>>> org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>>>         at
>>> org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>>>         - locked <0x0000000519111590> (a
>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>>         at org.apache.flume.channel.file.Log.take(Log.java:550)
>>>         at
>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>>>         at
>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>         at
>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>         at
>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>         at
>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>         at
>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>         at java.lang.Thread.run(Thread.java:662)
>>>
>>>
>>>
>>> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <
>>> hshreedharan@cloudera.com> wrote:
>>>
>>> Since the channel is designed to make sure that events are not
>>> duplicated to multiple sinks, and to protect against corruption due to
>>> concurrency issues, we do not need the locking in the channel's flume event
>>> queue. It is unlikely that locking is what is causing performance issues
>>> because the channel is heavily I/O bound. If you take a series of thread
>>> dumps, you will probably see that those threads are moving forward and the
>>> ones reading/writing from/to disk are the ones which are slower. These
>>> locks are unlikely to hit performance much.
>>>
>>> Thanks,
>>> Hari
>>>
>>> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
>>>
>>> Hi,
>>>
>>> Spent some more time debugging issues with FileChannel. The problem
>>> seems to lock contention reading from FlumeEventQueue:
>>>
>>> I see a lot of threads like this:
>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
>>> [0x00007f821afee000]
>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>         at
>>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>>>         - waiting to lock <0x0000000518ee4c90> (a
>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>         at
>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>>>         at
>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>         at
>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>         at
>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>         at
>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>         at
>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>         at java.lang.Thread.run(Thread.java:662)
>>>
>>>
>>> I have two file channels and 8 Avro Sinks per file channel. I added more
>>> sinks because they weren't draining fast enough. It seems like they send
>>> the batch then wait for an ack before sending again, thus sends are not
>>> pipelined and having more sinks seemed like a good way of getting some
>>> parallelism.
>>>
>>> Here's the full stack trace:
>>> 2013-08-13 15:30:32
>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02 mixed
>>> mode):
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
>>>    java.lang.Thread.State: WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000056d932120> (a
>>> java.util.concurrent.CountDownLatch$Sync)
>>>  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>>>  at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>>>  at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
>>> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
>>>  at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
>>> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
>>>  at
>>> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
>>> at $Proxy7.appendBatch(Unknown Source)
>>>  at
>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
>>> at
>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
>>>  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000 nid=0x41a1
>>> waiting on condition [0x0000000000000000]
>>>    java.lang.Thread.State: RUNNABLE
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>  at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005194c3340> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005191a16d8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519174e08> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519174e98> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005195f9670> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519174ce8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005191a1918> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005191e7f60> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153 runnable
>>> [0x00007f8210e4d000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
>>> - locked <0x000000054a2917a8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519174f58> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146 runnable
>>> [0x00007f8210f4e000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000546e5dd28> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118 runnable
>>> [0x00007f821104f000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000544f763c0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111 runnable
>>> [0x00007f8211150000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
>>> - locked <0x000000054473d5a8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110 runnable
>>> [0x00007f8211251000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005437f4f70> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f runnable
>>> [0x00007f8211352000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005422a9018> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e runnable
>>> [0x00007f8211453000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000541063d50> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d runnable
>>> [0x00007f8211554000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000540dc1110> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a runnable
>>> [0x00007f8211655000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
>>> - locked <0x000000053fbbc3b0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109 runnable
>>> [0x00007f8211756000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
>>> - locked <0x000000053f812860> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd runnable
>>> [0x00007f8211857000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
>>> - locked <0x000000053e6e5570> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519175138> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000519174c28> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005191751c8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0 runnable
>>> [0x00007f8211c5b000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000536579db0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x00000005191750a8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1 runnable
>>> [0x00007f8211e5d000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005313943a8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051936f460> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051936dfa8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de runnable
>>> [0x00007f8212160000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
>>> - locked <0x000000052dad4468> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd runnable
>>> [0x00007f8212261000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
>>> - locked <0x000000052c73c5f0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc runnable
>>> [0x00007f8212362000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
>>> - locked <0x000000052b8da998> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db waiting
>>> for monitor entry [0x00007f8212462000]
>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>> at
>>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>>> - waiting to lock <0x0000000518ee4c90> (a
>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>  at
>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>>> at
>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>>  at
>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>> at
>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>>  at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>  at
>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>>  at
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>  at
>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>  at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>>  at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>> at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>>  at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>> at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>  at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>> at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>>  at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>> at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>>  at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>  at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>  at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x000000051936df18> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>> tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000]
>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for  <0x0000000518fe7ae8> (a
>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>  at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>  at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8 runnable
>>> [0x00007f8212766000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005263d6ba8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7 runnable
>>> [0x00007f8212867000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000525d1d0e0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6 runnable
>>> [0x00007f8212968000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000525a689d8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5 runnable
>>> [0x00007f8212a69000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000524e1e7d8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4 runnable
>>> [0x00007f8212b6a000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000524d437a8> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-142-thread-36" prio=10 tid=0x00007f85843ad800 nid=0x40d3 runnable
>>> [0x00007f8212c6b000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x0000000524cf4140> (a sun.nio.ch.Util$2)
>>> - locked <0x0000000524cf4128> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000524ab32e8> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>  at java.lang.Thread.run(Thread.java:662)
>>>
>>> "pool-144-thread-35" prio=10 tid=0x00007f85d83a9000 nid=0x40d2 runnable
>>> [0x00007f8212d6c000]
>>>    java.lang.Thread.State: RUNNABLE
>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>  - locked <0x00000005244877c8> (a sun.nio.ch.Util$2)
>>> - locked <0x00000005244877b0> (a java.util.Collections$UnmodifiableSet)
>>>  - locked <0x0000000523f5b660> (a sun.nio.ch.EPollSelectorImpl)
>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>  at
>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>> at
>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>  at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>
>>> ...

Re: Lock contention in FileChannel

[Pankaj Gupta <pa...@brightroll.com>]

Here's the config:
# define channels, one for each disk




agent1.channels.ch1.type = FILE
agent1.channels.ch1.checkpointDir = /flume1/checkpoint
agent1.channels.ch1.dataDirs = /flume1/data
agent1.channels.ch1.maxFileSize = 375809638400
agent1.channels.ch1.capacity = 75000000
agent1.channels.ch1.transactionCapacity = 4000

agent1.channels.ch2.type = FILE
agent1.channels.ch2.checkpointDir = /flume2/checkpoint
agent1.channels.ch2.dataDirs = /flume2/data
agent1.channels.ch2.maxFileSize = 375809638400
agent1.channels.ch2.capacity = 75000000
agent1.channels.ch2.transactionCapacity = 4000



# Define an Avro source named avroSource1
# Each sink can connect to only one channel.
# Connect it to channel ch1. Load balance it to 2 avroSinks


agent1.sources.avroSource1.channels = ch1
agent1.sources.avroSource1.type = avro
agent1.sources.avroSource1.bind = 0.0.0.0
agent1.sources.avroSource1.port = <port>




agent1.sinks.avroSink1-1-1.type = avro
agent1.sinks.avroSink1-1-1.channel = ch1
agent1.sinks.avroSink1-1-1.hostname = <hostname>
agent1.sinks.avroSink1-1-1.port = <port>
agent1.sinks.avroSink1-1-1.connect-timeout = 300000
agent1.sinks.avroSink1-1-1.batch-size = 4000




agent1.sinks.avroSink1-2-1.type = avro
agent1.sinks.avroSink1-2-1.channel = ch1
agent1.sinks.avroSink1-2-1.hostname = <hostname>
agent1.sinks.avroSink1-2-1.port = <port>
agent1.sinks.avroSink1-2-1.connect-timeout = 300000
agent1.sinks.avroSink1-2-1.batch-size = 4000




agent1.sinks.avroSink1-3-1.type = avro
agent1.sinks.avroSink1-3-1.channel = ch1
agent1.sinks.avroSink1-3-1.hostname = <hostname>
agent1.sinks.avroSink1-3-1.port = <port>
agent1.sinks.avroSink1-3-1.connect-timeout = 300000
agent1.sinks.avroSink1-3-1.batch-size = 4000




agent1.sinks.avroSink1-4-1.type = avro
agent1.sinks.avroSink1-4-1.channel = ch1
agent1.sinks.avroSink1-4-1.hostname = <hostname>
agent1.sinks.avroSink1-4-1.port = <port>
agent1.sinks.avroSink1-4-1.connect-timeout = 300000
agent1.sinks.avroSink1-4-1.batch-size = 4000



#Add the sink groups; load-balance between each group of sinks which round
robin between different hops
agent1.sinkgroups.group1.sinks = avroSink1-1-1 avroSink1-2-1 avroSink1-3-1
avroSink1-4-1
agent1.sinkgroups.group1.processor.type = load_balance
agent1.sinkgroups.group1.processor.selector = ROUND_ROBIN
agent1.sinkgroups.group1.processor.backoff = true


#End of set

# Define an Avro source named avroSource2
# Each sink can connect to only one channel.
# Connect it to channel ch2. Load balance it to 2 avroSinks


agent1.sources.avroSource2.channels = ch2
agent1.sources.avroSource2.type = avro
agent1.sources.avroSource2.bind = 0.0.0.0
agent1.sources.avroSource2.port = <port>




agent1.sinks.avroSink2-1-1.type = avro
agent1.sinks.avroSink2-1-1.channel = ch2
agent1.sinks.avroSink2-1-1.hostname = <hostname>
agent1.sinks.avroSink2-1-1.port = <port>
agent1.sinks.avroSink2-1-1.connect-timeout = 300000
agent1.sinks.avroSink2-1-1.batch-size = 4000




agent1.sinks.avroSink2-2-1.type = avro
agent1.sinks.avroSink2-2-1.channel = ch2
agent1.sinks.avroSink2-2-1.hostname = <hostname>
agent1.sinks.avroSink2-2-1.port = <port>
agent1.sinks.avroSink2-2-1.connect-timeout = 300000
agent1.sinks.avroSink2-2-1.batch-size = 4000




agent1.sinks.avroSink2-3-1.type = avro
agent1.sinks.avroSink2-3-1.channel = ch2
agent1.sinks.avroSink2-3-1.hostname = <hostname>
agent1.sinks.avroSink2-3-1.port = <port>
agent1.sinks.avroSink2-3-1.connect-timeout = 300000
agent1.sinks.avroSink2-3-1.batch-size = 4000




agent1.sinks.avroSink2-4-1.type = avro
agent1.sinks.avroSink2-4-1.channel = ch2
agent1.sinks.avroSink2-4-1.hostname = <hostname>
agent1.sinks.avroSink2-4-1.port = <port>
agent1.sinks.avroSink2-4-1.connect-timeout = 300000
agent1.sinks.avroSink2-4-1.batch-size = 4000



#Add the sink groups; load-balance between each group of sinks which round
robin between different hops
agent1.sinkgroups.group2.sinks = avroSink2-1-1 avroSink2-2-1 avroSink2-3-1
avroSink2-4-1
agent1.sinkgroups.group2.processor.type = load_balance
agent1.sinkgroups.group2.processor.selector = ROUND_ROBIN
agent1.sinkgroups.group2.processor.backoff = true


#End of set

# Define an Avro source named avroSource3
# Each sink can connect to only one channel.
# Connect it to channel ch1. Load balance it to 2 avroSinks


agent1.sources.avroSource3.channels = ch1
agent1.sources.avroSource3.type = avro
agent1.sources.avroSource3.bind = 0.0.0.0
agent1.sources.avroSource3.port = <port>




agent1.sinks.avroSink3-1-1.type = avro
agent1.sinks.avroSink3-1-1.channel = ch1
agent1.sinks.avroSink3-1-1.hostname = <hostname>
agent1.sinks.avroSink3-1-1.port = <port>
agent1.sinks.avroSink3-1-1.connect-timeout = 300000
agent1.sinks.avroSink3-1-1.batch-size = 4000




agent1.sinks.avroSink3-2-1.type = avro
agent1.sinks.avroSink3-2-1.channel = ch1
agent1.sinks.avroSink3-2-1.hostname = <hostname>
agent1.sinks.avroSink3-2-1.port = <port>
agent1.sinks.avroSink3-2-1.connect-timeout = 300000
agent1.sinks.avroSink3-2-1.batch-size = 4000




agent1.sinks.avroSink3-3-1.type = avro
agent1.sinks.avroSink3-3-1.channel = ch1
agent1.sinks.avroSink3-3-1.hostname = <hostname>
agent1.sinks.avroSink3-3-1.port = <port>
agent1.sinks.avroSink3-3-1.connect-timeout = 300000
agent1.sinks.avroSink3-3-1.batch-size = 4000




agent1.sinks.avroSink3-4-1.type = avro
agent1.sinks.avroSink3-4-1.channel = ch1
agent1.sinks.avroSink3-4-1.hostname = <hostname>
agent1.sinks.avroSink3-4-1.port = <port>
agent1.sinks.avroSink3-4-1.connect-timeout = 300000
agent1.sinks.avroSink3-4-1.batch-size = 4000



#Add the sink groups; load-balance between each group of sinks which round
robin between different hops
agent1.sinkgroups.group3.sinks = avroSink3-1-1 avroSink3-2-1 avroSink3-3-1
avroSink3-4-1
agent1.sinkgroups.group3.processor.type = load_balance
agent1.sinkgroups.group3.processor.selector = ROUND_ROBIN
agent1.sinkgroups.group3.processor.backoff = true


#End of set

# Define an Avro source named avroSource4
# Each sink can connect to only one channel.
# Connect it to channel ch2. Load balance it to 2 avroSinks


agent1.sources.avroSource4.channels = ch2
agent1.sources.avroSource4.type = avro
agent1.sources.avroSource4.bind = 0.0.0.0
agent1.sources.avroSource4.port = <port>




agent1.sinks.avroSink4-1-1.type = avro
agent1.sinks.avroSink4-1-1.channel = ch2
agent1.sinks.avroSink4-1-1.hostname = <hostname>
agent1.sinks.avroSink4-1-1.port = <port>
agent1.sinks.avroSink4-1-1.connect-timeout = 300000
agent1.sinks.avroSink4-1-1.batch-size = 4000




agent1.sinks.avroSink4-2-1.type = avro
agent1.sinks.avroSink4-2-1.channel = ch2
agent1.sinks.avroSink4-2-1.hostname = <hostname>
agent1.sinks.avroSink4-2-1.port = <port>
agent1.sinks.avroSink4-2-1.connect-timeout = 300000
agent1.sinks.avroSink4-2-1.batch-size = 4000




agent1.sinks.avroSink4-3-1.type = avro
agent1.sinks.avroSink4-3-1.channel = ch2
agent1.sinks.avroSink4-3-1.hostname = <hostname>
agent1.sinks.avroSink4-3-1.port = <port>
agent1.sinks.avroSink4-3-1.connect-timeout = 300000
agent1.sinks.avroSink4-3-1.batch-size = 4000




agent1.sinks.avroSink4-4-1.type = avro
agent1.sinks.avroSink4-4-1.channel = ch2
agent1.sinks.avroSink4-4-1.hostname = <hostname>
agent1.sinks.avroSink4-4-1.port = <port>
agent1.sinks.avroSink4-4-1.connect-timeout = 300000
agent1.sinks.avroSink4-4-1.batch-size = 4000



#Add the sink groups; load-balance between each group of sinks which round
robin between different hops
agent1.sinkgroups.group4.sinks = avroSink4-1-1 avroSink4-2-1 avroSink4-3-1
avroSink4-4-1
agent1.sinkgroups.group4.processor.type = load_balance
agent1.sinkgroups.group4.processor.selector = ROUND_ROBIN
agent1.sinkgroups.group4.processor.backoff = true


#End of set



# Finally, now that we've defined all of our components, tell
# agent1 which ones we want to activate.
agent1.channels = ch1 ch2
agent1.sources = avroSource1 avroSource2 avroSource3 avroSource4
agent1.sinks = avroSink1-1-1 avroSink1-2-1 avroSink1-3-1 avroSink1-4-1
avroSink2-1-1 avroSink2-2-1 avroSink2-3-1 avroSink2-4-1 avroSink3-1-1
avroSink3-2-1 avroSink3-3-1 avroSink3-4-1 avroSink4-1-1 avroSink4-2-1
avroSink4-3-1 avroSink4-4-1
agent1.sinkgroups = group1 group2 group3 group4

There are two disks on the box /flume1 and /flume2
32 cores of CPU, 32 GB RAM, 1GbPS ethernet.


On Tue, Aug 13, 2013 at 7:22 PM, Brock Noland <br...@cloudera.com> wrote:

> Can you share your conf file?
> On Aug 13, 2013 9:19 PM, "Hari Shreedharan" <hs...@cloudera.com>
> wrote:
>
>>  Even though the writes are done per batch, they don't go to disk
>> rightaway - commits are the only ones which actually cause an fsync - which
>> is when writes actually go to disk.
>>
>>
>> Thanks,
>> Hari
>>
>> On Tuesday, August 13, 2013 at 7:06 PM, Pankaj Gupta wrote:
>>
>> Looking at the code it seems like the lock and the i/o is done per event
>> and not for a batch. Is that correct? If that is the case then it seems
>> like there is a lot of overhead per event. The throughput I'm seeing is 1 -
>> 1.5 MBps per disk which is way below the sequential read/write capacity of
>> the disk which is easily over 50MBps. Adding more sinks doesn't help, they
>> just block waiting for the queue to become free. CPU usage is 20%, there is
>> enough RAM for page cache so that no read is going to disk. The queue seems
>> to be the bottleneck. What is the throughput I should expect per disk?
>>
>>
>> On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland <br...@cloudera.com> wrote:
>>
>> The lock is per file. Adding more directories to the channel will cause
>> more files to be created. Of course you'll need additional disks behind
>> those directories to see any performance increase.
>>
>>
>> On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <
>> hshreedharan@cloudera.com> wrote:
>>
>>  Yes, IO is done inside locks to avoid multiple takes and puts getting
>> written out at the same time. Even though Java makes sure the writes are
>> serialized, Flume still needs to keep track of some counters etc, so the
>> lock is required. Note that the lock you are talking about is  in the
>> LogFile class, which represents a single file - so even if the write is
>> inside that lock (which is also inside that class itself) that  does not
>> cause any contention - because the lock is just preventing two IO ops to
>> happen at the same time.
>>
>>
>> Thanks,
>> Hari
>>
>> On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:
>>
>> It seems like some i/o is done inside the lock, which means that time for
>> taking a lock is proportional to the time for i/o and thus it becomes a
>> problem. I apologize in advance if I am wrong but the call stack and
>> behavior I'm seeing seems to suggest that. Specifically, it seems that we
>> do a write while inside take:
>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.NativeThread.current(Native Method)
>>         at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>>         at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>>         - locked <0x00000005190ec998> (a java.lang.Object)
>>         at
>> org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>>         at
>> org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>>         - locked <0x0000000519111590> (a
>> org.apache.flume.channel.file.LogFileV3$Writer)
>>         at org.apache.flume.channel.file.Log.take(Log.java:550)
>>         at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>>         at
>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>         at
>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>         at
>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>         at
>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>         at
>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>         at java.lang.Thread.run(Thread.java:662)
>>
>>
>>
>> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <
>> hshreedharan@cloudera.com> wrote:
>>
>> Since the channel is designed to make sure that events are not duplicated
>> to multiple sinks, and to protect against corruption due to concurrency
>> issues, we do not need the locking in the channel's flume event queue. It
>> is unlikely that locking is what is causing performance issues because the
>> channel is heavily I/O bound. If you take a series of thread dumps, you
>> will probably see that those threads are moving forward and the ones
>> reading/writing from/to disk are the ones which are slower. These locks are
>> unlikely to hit performance much.
>>
>> Thanks,
>> Hari
>>
>> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
>>
>> Hi,
>>
>> Spent some more time debugging issues with FileChannel. The problem seems
>> to lock contention reading from FlumeEventQueue:
>>
>> I see a lot of threads like this:
>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
>> [0x00007f821afee000]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>         at
>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>>         - waiting to lock <0x0000000518ee4c90> (a
>> org.apache.flume.channel.file.FlumeEventQueue)
>>         at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>>         at
>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>         at
>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>         at
>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>         at
>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>         at
>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>         at java.lang.Thread.run(Thread.java:662)
>>
>>
>> I have two file channels and 8 Avro Sinks per file channel. I added more
>> sinks because they weren't draining fast enough. It seems like they send
>> the batch then wait for an ack before sending again, thus sends are not
>> pipelined and having more sinks seemed like a good way of getting some
>> parallelism.
>>
>> Here's the full stack trace:
>> 2013-08-13 15:30:32
>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02 mixed mode):
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000056d932120> (a
>> java.util.concurrent.CountDownLatch$Sync)
>>  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>>  at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>>  at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
>> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
>>  at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
>> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
>>  at
>> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
>> at $Proxy7.appendBatch(Unknown Source)
>>  at
>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
>> at
>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
>>  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000 nid=0x41a1
>> waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>  at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005194c3340> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005191a16d8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519174e08> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519174e98> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005195f9670> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519174ce8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005191a1918> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005191e7f60> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153 runnable
>> [0x00007f8210e4d000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
>> - locked <0x000000054a2917a8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519174f58> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146 runnable
>> [0x00007f8210f4e000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
>> - locked <0x0000000546e5dd28> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118 runnable
>> [0x00007f821104f000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000544f763c0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111 runnable
>> [0x00007f8211150000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
>> - locked <0x000000054473d5a8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110 runnable
>> [0x00007f8211251000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
>> - locked <0x00000005437f4f70> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f runnable
>> [0x00007f8211352000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
>> - locked <0x00000005422a9018> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e runnable
>> [0x00007f8211453000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
>> - locked <0x0000000541063d50> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d runnable
>> [0x00007f8211554000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
>> - locked <0x0000000540dc1110> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a runnable
>> [0x00007f8211655000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
>> - locked <0x000000053fbbc3b0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109 runnable
>> [0x00007f8211756000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
>> - locked <0x000000053f812860> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd runnable
>> [0x00007f8211857000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
>> - locked <0x000000053e6e5570> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519175138> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519174c28> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005191751c8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0 runnable
>> [0x00007f8211c5b000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000536579db0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005191750a8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1 runnable
>> [0x00007f8211e5d000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
>> - locked <0x00000005313943a8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051936f460> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051936dfa8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de runnable
>> [0x00007f8212160000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
>> - locked <0x000000052dad4468> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd runnable
>> [0x00007f8212261000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
>> - locked <0x000000052c73c5f0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc runnable
>> [0x00007f8212362000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
>> - locked <0x000000052b8da998> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db waiting
>> for monitor entry [0x00007f8212462000]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>> - waiting to lock <0x0000000518ee4c90> (a
>> org.apache.flume.channel.file.FlumeEventQueue)
>>  at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>> at
>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>  at
>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>> at
>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>  at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>>  at
>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>  at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>  at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>  at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>  at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>  at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>  at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051936df18> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000518fe7ae8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8 runnable
>> [0x00007f8212766000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
>> - locked <0x00000005263d6ba8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7 runnable
>> [0x00007f8212867000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000525d1d0e0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6 runnable
>> [0x00007f8212968000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000525a689d8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5 runnable
>> [0x00007f8212a69000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000524e1e7d8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4 runnable
>> [0x00007f8212b6a000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000524d437a8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-36" prio=10 tid=0x00007f85843ad800 nid=0x40d3 runnable
>> [0x00007f8212c6b000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000524cf4140> (a sun.nio.ch.Util$2)
>> - locked <0x0000000524cf4128> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000524ab32e8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-35" prio=10 tid=0x00007f85d83a9000 nid=0x40d2 runnable
>> [0x00007f8212d6c000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005244877c8> (a sun.nio.ch.Util$2)
>> - locked <0x00000005244877b0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000523f5b660> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-34" prio=10 tid=0x00007f85d838e000 nid=0x40d1 runnable
>> [0x00007f8212e6d000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000052419e490> (a sun.nio.ch.Util$2)
>> - locked <0x000000052419e478> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000523f5b6f0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-34" prio=10 tid=0x00007f8588390800 nid=0x40d0 waiting
>> for monitor entry [0x00007f8212f6d000]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
>> - waiting to lock <0x0000000519111590> (a
>> org.apache.flume.channel.file.LogFileV3$Writer)
>>  at org.apache.flume.channel.file.Log.put(Log.java:507)
>> at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
>>  at
>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>> at
>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>  at
>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>  at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>  at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>> at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>  at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>> at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>  at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>  at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>  at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>  at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-35" prio=10 tid=0x00007f85843ab800 nid=0x40cf runnable
>> [0x00007f821306f000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000523d40938> (a sun.nio.ch.Util$2)
>> - locked <0x0000000523d40aa0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000523ecec18> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-37" prio=10 tid=0x00007f85803e3800 nid=0x40ce waiting
>> for monitor entry [0x00007f821316f000]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>> - waiting to lock <0x0000000518ee4c90> (a
>> org.apache.flume.channel.file.FlumeEventQueue)
>>  at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>> at
>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>  at
>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>> at
>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>  at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>>  at
>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>  at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>  at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>  at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>  at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>
>> ...
>
>


-- 


*P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com

Pankaj Gupta | Software Engineer

*BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com


United States | Canada | United Kingdom | Germany


We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
!

Re: Lock contention in FileChannel

[Brock Noland <br...@cloudera.com>]

Can you share your conf file?
On Aug 13, 2013 9:19 PM, "Hari Shreedharan" <hs...@cloudera.com>
wrote:

>  Even though the writes are done per batch, they don't go to disk
> rightaway - commits are the only ones which actually cause an fsync - which
> is when writes actually go to disk.
>
>
> Thanks,
> Hari
>
> On Tuesday, August 13, 2013 at 7:06 PM, Pankaj Gupta wrote:
>
> Looking at the code it seems like the lock and the i/o is done per event
> and not for a batch. Is that correct? If that is the case then it seems
> like there is a lot of overhead per event. The throughput I'm seeing is 1 -
> 1.5 MBps per disk which is way below the sequential read/write capacity of
> the disk which is easily over 50MBps. Adding more sinks doesn't help, they
> just block waiting for the queue to become free. CPU usage is 20%, there is
> enough RAM for page cache so that no read is going to disk. The queue seems
> to be the bottleneck. What is the throughput I should expect per disk?
>
>
> On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland <br...@cloudera.com> wrote:
>
> The lock is per file. Adding more directories to the channel will cause
> more files to be created. Of course you'll need additional disks behind
> those directories to see any performance increase.
>
>
> On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <
> hshreedharan@cloudera.com> wrote:
>
>  Yes, IO is done inside locks to avoid multiple takes and puts getting
> written out at the same time. Even though Java makes sure the writes are
> serialized, Flume still needs to keep track of some counters etc, so the
> lock is required. Note that the lock you are talking about is  in the
> LogFile class, which represents a single file - so even if the write is
> inside that lock (which is also inside that class itself) that  does not
> cause any contention - because the lock is just preventing two IO ops to
> happen at the same time.
>
>
> Thanks,
> Hari
>
> On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:
>
> It seems like some i/o is done inside the lock, which means that time for
> taking a lock is proportional to the time for i/o and thus it becomes a
> problem. I apologize in advance if I am wrong but the call stack and
> behavior I'm seeing seems to suggest that. Specifically, it seems that we
> do a write while inside take:
> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.NativeThread.current(Native Method)
>         at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>         at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>         - locked <0x00000005190ec998> (a java.lang.Object)
>         at
> org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>         at
> org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>         - locked <0x0000000519111590> (a
> org.apache.flume.channel.file.LogFileV3$Writer)
>         at org.apache.flume.channel.file.Log.take(Log.java:550)
>         at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>         at
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>         at
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>         at
> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>         at
> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:662)
>
>
>
> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <
> hshreedharan@cloudera.com> wrote:
>
> Since the channel is designed to make sure that events are not duplicated
> to multiple sinks, and to protect against corruption due to concurrency
> issues, we do not need the locking in the channel's flume event queue. It
> is unlikely that locking is what is causing performance issues because the
> channel is heavily I/O bound. If you take a series of thread dumps, you
> will probably see that those threads are moving forward and the ones
> reading/writing from/to disk are the ones which are slower. These locks are
> unlikely to hit performance much.
>
> Thanks,
> Hari
>
> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
>
> Hi,
>
> Spent some more time debugging issues with FileChannel. The problem seems
> to lock contention reading from FlumeEventQueue:
>
> I see a lot of threads like this:
> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
> [0x00007f821afee000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at
> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>         - waiting to lock <0x0000000518ee4c90> (a
> org.apache.flume.channel.file.FlumeEventQueue)
>         at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>         at
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>         at
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>         at
> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>         at
> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:662)
>
>
> I have two file channels and 8 Avro Sinks per file channel. I added more
> sinks because they weren't draining fast enough. It seems like they send
> the batch then wait for an ack before sending again, thus sends are not
> pipelined and having more sinks seemed like a good way of getting some
> parallelism.
>
> Here's the full stack trace:
> 2013-08-13 15:30:32
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02 mixed mode):
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000056d932120> (a
> java.util.concurrent.CountDownLatch$Sync)
>  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>  at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>  at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
>  at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
>  at
> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
> at $Proxy7.appendBatch(Unknown Source)
>  at
> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
> at
> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
>  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000 nid=0x41a1 waiting
> on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>  at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005194c3340> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191a16d8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519174e08> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519174e98> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005195f9670> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519174ce8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191a1918> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191e7f60> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153 runnable
> [0x00007f8210e4d000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
> - locked <0x000000054a2917a8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519174f58> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146 runnable
> [0x00007f8210f4e000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
> - locked <0x0000000546e5dd28> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118 runnable
> [0x00007f821104f000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
> - locked <0x0000000544f763c0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111 runnable
> [0x00007f8211150000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
> - locked <0x000000054473d5a8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110 runnable
> [0x00007f8211251000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
> - locked <0x00000005437f4f70> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f runnable
> [0x00007f8211352000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
> - locked <0x00000005422a9018> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e runnable
> [0x00007f8211453000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
> - locked <0x0000000541063d50> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d runnable
> [0x00007f8211554000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
> - locked <0x0000000540dc1110> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a runnable
> [0x00007f8211655000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
> - locked <0x000000053fbbc3b0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109 runnable
> [0x00007f8211756000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
> - locked <0x000000053f812860> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd runnable
> [0x00007f8211857000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
> - locked <0x000000053e6e5570> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519175138> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519174c28> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191751c8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0 runnable
> [0x00007f8211c5b000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
> - locked <0x0000000536579db0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191750a8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1 runnable
> [0x00007f8211e5d000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
> - locked <0x00000005313943a8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051936f460> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051936dfa8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de runnable
> [0x00007f8212160000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
> - locked <0x000000052dad4468> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd runnable
> [0x00007f8212261000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
> - locked <0x000000052c73c5f0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc runnable
> [0x00007f8212362000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
> - locked <0x000000052b8da998> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db waiting for
> monitor entry [0x00007f8212462000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
> - waiting to lock <0x0000000518ee4c90> (a
> org.apache.flume.channel.file.FlumeEventQueue)
>  at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
> at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>  at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>  at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>  at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051936df18> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518fe7ae8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8 runnable
> [0x00007f8212766000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
> - locked <0x00000005263d6ba8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7 runnable
> [0x00007f8212867000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
> - locked <0x0000000525d1d0e0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6 runnable
> [0x00007f8212968000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
> - locked <0x0000000525a689d8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5 runnable
> [0x00007f8212a69000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
> - locked <0x0000000524e1e7d8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4 runnable
> [0x00007f8212b6a000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
> - locked <0x0000000524d437a8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-36" prio=10 tid=0x00007f85843ad800 nid=0x40d3 runnable
> [0x00007f8212c6b000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000524cf4140> (a sun.nio.ch.Util$2)
> - locked <0x0000000524cf4128> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000524ab32e8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-35" prio=10 tid=0x00007f85d83a9000 nid=0x40d2 runnable
> [0x00007f8212d6c000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005244877c8> (a sun.nio.ch.Util$2)
> - locked <0x00000005244877b0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000523f5b660> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-34" prio=10 tid=0x00007f85d838e000 nid=0x40d1 runnable
> [0x00007f8212e6d000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052419e490> (a sun.nio.ch.Util$2)
> - locked <0x000000052419e478> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000523f5b6f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-34" prio=10 tid=0x00007f8588390800 nid=0x40d0 waiting for
> monitor entry [0x00007f8212f6d000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
> - waiting to lock <0x0000000519111590> (a
> org.apache.flume.channel.file.LogFileV3$Writer)
>  at org.apache.flume.channel.file.Log.put(Log.java:507)
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
>  at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>  at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>  at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>  at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>  at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-35" prio=10 tid=0x00007f85843ab800 nid=0x40cf runnable
> [0x00007f821306f000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000523d40938> (a sun.nio.ch.Util$2)
> - locked <0x0000000523d40aa0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000523ecec18> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-37" prio=10 tid=0x00007f85803e3800 nid=0x40ce waiting for
> monitor entry [0x00007f821316f000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
> - waiting to lock <0x0000000518ee4c90> (a
> org.apache.flume.channel.file.FlumeEventQueue)
>  at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
> at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>  at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>  at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>
> ...

Re: Lock contention in FileChannel

[Hari Shreedharan <hs...@cloudera.com>]

Even though the writes are done per batch, they don't go to disk rightaway - commits are the only ones which actually cause an fsync - which is when writes actually go to disk.  


Thanks,
Hari


On Tuesday, August 13, 2013 at 7:06 PM, Pankaj Gupta wrote:

> Looking at the code it seems like the lock and the i/o is done per event and not for a batch. Is that correct? If that is the case then it seems like there is a lot of overhead per event. The throughput I'm seeing is 1 - 1.5 MBps per disk which is way below the sequential read/write capacity of the disk which is easily over 50MBps. Adding more sinks doesn't help, they just block waiting for the queue to become free. CPU usage is 20%, there is enough RAM for page cache so that no read is going to disk. The queue seems to be the bottleneck. What is the throughput I should expect per disk? 
> 
> 
> On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland <brock@cloudera.com (mailto:brock@cloudera.com)> wrote:
> > The lock is per file. Adding more directories to the channel will cause more files to be created. Of course you'll need additional disks behind those directories to see any performance increase. 
> > 
> > 
> > On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <hshreedharan@cloudera.com (mailto:hshreedharan@cloudera.com)> wrote:
> > > Yes, IO is done inside locks to avoid multiple takes and puts getting written out at the same time. Even though Java makes sure the writes are serialized, Flume still needs to keep track of some counters etc, so the lock is required. Note that the lock you are talking about is  in the LogFile class, which represents a single file - so even if the write is inside that lock (which is also inside that class itself) that  does not cause any contention - because the lock is just preventing two IO ops to happen at the same time. 
> > > 
> > > 
> > > Thanks,
> > > Hari
> > > 
> > > 
> > > On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:
> > > 
> > > > It seems like some i/o is done inside the lock, which means that time for taking a lock is proportional to the time for i/o and thus it becomes a problem. I apologize in advance if I am wrong but the call stack and behavior I'm seeing seems to suggest that. Specifically, it seems that we do a write while inside take:
> > > > "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10 tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
> > > >    java.lang.Thread.State: RUNNABLE
> > > >         at sun.nio.ch.NativeThread.current(Native Method)
> > > >         at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
> > > >         at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
> > > >         - locked <0x00000005190ec998> (a java.lang.Object)
> > > >         at org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
> > > >         at org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
> > > >         - locked <0x0000000519111590> (a org.apache.flume.channel.file.LogFileV3$Writer)
> > > >         at org.apache.flume.channel.file.Log.take(Log.java:550)
> > > >         at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
> > > >         at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> > > >         at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> > > >         at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
> > > >         at org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> > > >         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> > > >         at java.lang.Thread.run(Thread.java:662)
> > > > 
> > > > 
> > > > 
> > > > On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <hshreedharan@cloudera.com (mailto:hshreedharan@cloudera.com)> wrote:
> > > > > Since the channel is designed to make sure that events are not duplicated to multiple sinks, and to protect against corruption due to concurrency issues, we do not need the locking in the channel's flume event queue. It is unlikely that locking is what is causing performance issues because the channel is heavily I/O bound. If you take a series of thread dumps, you will probably see that those threads are moving forward and the ones reading/writing from/to disk are the ones which are slower. These locks are unlikely to hit performance much. 
> > > > > 
> > > > > Thanks,
> > > > > Hari
> > > > > 
> > > > > 
> > > > > On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
> > > > > 
> > > > > > Hi,
> > > > > > 
> > > > > > Spent some more time debugging issues with FileChannel. The problem seems to lock contention reading from FlumeEventQueue:
> > > > > > 
> > > > > > I see a lot of threads like this: 
> > > > > > "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10 tid=0x00007f857b378800 nid=0x404d waiting for monitor entry [0x00007f821afee000]
> > > > > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > > > >         at org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
> > > > > >         - waiting to lock <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> > > > > >         at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
> > > > > >         at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> > > > > >         at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> > > > > >         at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
> > > > > >         at org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> > > > > >         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> > > > > >         at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > 
> > > > > > I have two file channels and 8 Avro Sinks per file channel. I added more sinks because they weren't draining fast enough. It seems like they send the batch then wait for an ack before sending again, thus sends are not pipelined and having more sinks seemed like a good way of getting some parallelism. 
> > > > > > 
> > > > > > Here's the full stack trace:
> > > > > > 2013-08-13 15:30:32
> > > > > > Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02 mixed mode):
> > > > > > 
> > > > > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000] 
> > > > > >    java.lang.Thread.State: WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000056d932120> (a java.util.concurrent.CountDownLatch$Sync)
> > > > > > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
> > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
> > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
> > > > > > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
> > > > > > at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
> > > > > > at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
> > > > > > at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
> > > > > > at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
> > > > > > at $Proxy7.appendBatch(Unknown Source)
> > > > > > at org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
> > > > > > at org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
> > > > > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Attach Listener" daemon prio=10 tid=0x00007f85fc016000 nid=0x41a1 waiting on condition [0x0000000000000000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > 
> > > > > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005194c3340> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005191a16d8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519174e08> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519174e98> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005195f9670> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519174ce8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005191a1918> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005191e7f60> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153 runnable [0x00007f8210e4d000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000054a2917a8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519174f58> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146 runnable [0x00007f8210f4e000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000546e5dd28> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118 runnable [0x00007f821104f000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000544f763c0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111 runnable [0x00007f8211150000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000054473d5a8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110 runnable [0x00007f8211251000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005437f4f70> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f runnable [0x00007f8211352000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005422a9018> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e runnable [0x00007f8211453000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000541063d50> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d runnable [0x00007f8211554000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000540dc1110> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a runnable [0x00007f8211655000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000053fbbc3b0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109 runnable [0x00007f8211756000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000053f812860> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd runnable [0x00007f8211857000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000053e6e5570> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519175138> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519174c28> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005191751c8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0 runnable [0x00007f8211c5b000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000536579db0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005191750a8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1 runnable [0x00007f8211e5d000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005313943a8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051936f460> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051936dfa8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de runnable [0x00007f8212160000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000052dad4468> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd runnable [0x00007f8212261000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000052c73c5f0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc runnable [0x00007f8212362000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000052b8da998> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db waiting for monitor entry [0x00007f8212462000] 
> > > > > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > > > > at org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
> > > > > > - waiting to lock <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> > > > > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
> > > > > > at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> > > > > > at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> > > > > > at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> > > > > > at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> > > > > > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > > > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > > > > at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> > > > > > at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> > > > > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> > > > > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > > > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> > > > > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051936df18> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000518fe7ae8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8 runnable [0x00007f8212766000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005263d6ba8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7 runnable [0x00007f8212867000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000525d1d0e0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6 runnable [0x00007f8212968000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000525a689d8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5 runnable [0x00007f8212a69000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000524e1e7d8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4 runnable [0x00007f8212b6a000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000524d437a8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-36" prio=10 tid=0x00007f85843ad800 nid=0x40d3 runnable [0x00007f8212c6b000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000524cf4140> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000524cf4128> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000524ab32e8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-35" prio=10 tid=0x00007f85d83a9000 nid=0x40d2 runnable [0x00007f8212d6c000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005244877c8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005244877b0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000523f5b660> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-34" prio=10 tid=0x00007f85d838e000 nid=0x40d1 runnable [0x00007f8212e6d000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000052419e490> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000052419e478> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000523f5b6f0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-34" prio=10 tid=0x00007f8588390800 nid=0x40d0 waiting for monitor entry [0x00007f8212f6d000] 
> > > > > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > > > > at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
> > > > > > - waiting to lock <0x0000000519111590> (a org.apache.flume.channel.file.LogFileV3$Writer)
> > > > > > at org.apache.flume.channel.file.Log.put(Log.java:507)
> > > > > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
> > > > > > at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> > > > > > at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> > > > > > at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> > > > > > at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> > > > > > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > > > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > > > > at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> > > > > > at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> > > > > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> > > > > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > > > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> > > > > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-35" prio=10 tid=0x00007f85843ab800 nid=0x40cf runnable [0x00007f821306f000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000523d40938> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000523d40aa0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000523ecec18> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-37" prio=10 tid=0x00007f85803e3800 nid=0x40ce waiting for monitor entry [0x00007f821316f000] 
> > > > > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > > > > at org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
> > > > > > - waiting to lock <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> > > > > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
> > > > > > at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> > > > > > at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> > > > > > at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> > > > > > at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> > > > > > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > > > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > > > > at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> > > > > > at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> > > > > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> > > > > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > > > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> > > > > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-34" prio=10 tid=0x00007f8584391000 nid=0x40cd runnable [0x00007f8213271000]
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000523935540> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000523935528> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000523788ed8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-33" prio=10 tid=0x00007f85d8373800 nid=0x40cc runnable [0x00007f8213372000]
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005239ec658> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005239ec640> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000523845410> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-33" prio=10 tid=0x00007f8588375800 nid=0x40cb waiting for monitor entry [0x00007f8213472000]
> > > > > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > > > > at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
> > > > > > - waiting to lock <0x0000000519111590> (a org.apache.flume.channel.file.LogFileV3$Writer)
> > > > > > at org.apache.flume.channel.file.Log.put(Log.java:507)
> > > > > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
> > > > > > at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> > > > > > at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> > > > > > at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> > > > > > at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> > > > > > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > > > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > > > > at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> > > > > > at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> > > > > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> > > > > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > > > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> > > > > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-33" prio=10 tid=0x00007f8584376000 nid=0x40ca runnable [0x00007f8213574000]
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005237e3028> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005237e3010> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000523788e90> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-32" prio=10 tid=0x00007f85d8358800 nid=0x40c9 waiting for monitor entry [0x00007f8213674000]
> > > > > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > > > > at org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
> > > > > > - waiting to lock <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> > > > > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
> > > > > > at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> > > > > > at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> > > > > > at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> > > > > > at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> > > > > > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > > > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > > > > at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> > > > > > at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> > > > > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> > > > > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > > > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> > > > > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-32" prio=10 tid=0x00007f858835b000 nid=0x40c8 runnable [0x00007f8213776000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000523343688> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000523343670> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000052305e5c8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-36" prio=10 tid=0x00007f85803c9000 nid=0x40c7 runnable [0x00007f8213877000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005230fa580> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005230fa568> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000522f5ca50> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-32" prio=10 tid=0x00007f858435b000 nid=0x40c6 runnable [0x00007f8213978000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005231367a8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000523136790> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000052305e658> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-31" prio=10 tid=0x00007f85d833e000 nid=0x40c5 runnable [0x00007f8213a79000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000522db30a0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000522db3088> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000522bda430> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-31" prio=10 tid=0x00007f8588340000 nid=0x40c4 runnable [0x00007f8213b7a000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000522c16518> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000522c16500> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000522bd97e8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-30" prio=10 tid=0x00007f85d8323800 nid=0x40c3 runnable [0x00007f8213c7b000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000522bd1a20> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000522bd1a08> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000522bda3a0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-31" prio=10 tid=0x00007f8584340800 nid=0x40c2 runnable [0x00007f8213d7c000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005228d1608> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005228d15f0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005228a6aa0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-30" prio=10 tid=0x00007f8588325000 nid=0x40c1 runnable [0x00007f8213e7d000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000052277b088> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000052277b070> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005225ed3b8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-35" prio=10 tid=0x00007f85803ae000 nid=0x40c0 runnable [0x00007f8213f7e000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000522737380> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000522737368> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005225ed328> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-29" prio=10 tid=0x00007f85d8308800 nid=0x40bf runnable [0x00007f821407f000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005226034a8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000522603490> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005225ed298> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-28" prio=10 tid=0x00007f85d82ed800 nid=0x40be runnable [0x00007f8214180000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000522359408> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005223593f0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005221e7278> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-27" prio=10 tid=0x00007f85d82d2800 nid=0x40bd runnable [0x00007f8214281000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005222c0478> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005222c0460> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005221e6e40> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-29" prio=10 tid=0x00007f858830a800 nid=0x40bc runnable [0x00007f8214382000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000521ddf488> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000521ddf470> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000521c88d38> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-30" prio=10 tid=0x00007f8584326000 nid=0x40bb waiting for monitor entry [0x00007f8214482000] 
> > > > > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > > > > at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
> > > > > > - waiting to lock <0x0000000519111590> (a org.apache.flume.channel.file.LogFileV3$Writer)
> > > > > > at org.apache.flume.channel.file.Log.put(Log.java:507)
> > > > > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
> > > > > > at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> > > > > > at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> > > > > > at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> > > > > > at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> > > > > > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > > > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > > > > at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> > > > > > at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> > > > > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> > > > > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > > > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> > > > > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-26" prio=10 tid=0x00007f85d82b8000 nid=0x40ba runnable [0x00007f8214584000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000521cbcb90> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000521cbcb78> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000521279250> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-28" prio=10 tid=0x00007f85882ef800 nid=0x40b9 runnable [0x00007f8214685000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000521a1c2d0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000521a1c2b8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005218f1508> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-34" prio=10 tid=0x00007f8580393800 nid=0x40b8 runnable [0x00007f8214786000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000052181ce58> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000052181ce40> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005212791c0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-29" prio=10 tid=0x00007f858430b800 nid=0x40b7 runnable [0x00007f8214887000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005219175f0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000521746090> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000521278e60> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-27" prio=10 tid=0x00007f85882d4800 nid=0x40b6 runnable [0x00007f8214988000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000052163e9c8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000052163e9b0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000521279130> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-33" prio=10 tid=0x00007f8580378800 nid=0x40b5 runnable [0x00007f8214a89000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000052152cb38> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000052152cb20> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000521278d40> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-28" prio=10 tid=0x00007f85842f0800 nid=0x40b4 runnable [0x00007f8214b8a000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005213ba3e0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005213ba3c8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000521278020> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-27" prio=10 tid=0x00007f85842d5800 nid=0x40b3 runnable [0x00007f8214c8b000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005213ed660> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005213ed648> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000521278c20> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-26" prio=10 tid=0x00007f85882b9800 nid=0x40b2 runnable [0x00007f8214d8c000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000052133b7c0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000052133b7a8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005212790a0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-32" prio=10 tid=0x00007f858035d800 nid=0x40af runnable [0x00007f8214e8d000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000520e53be0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000520e53bc8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000520f3bb50> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-25" prio=10 tid=0x00007f85d829d000 nid=0x40ae runnable [0x00007f8214f8e000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000520e09a30> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000520e09a18> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000520f3b6d0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-31" prio=10 tid=0x00007f8580342800 nid=0x40ad runnable [0x00007f821508f000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000520ef70e0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000520ef70c8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000520f3b208> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-26" prio=10 tid=0x00007f85842ba800 nid=0x40ac runnable [0x00007f8215190000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000520c9dfc0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000520c9dfa8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000520bb3cc0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-25" prio=10 tid=0x00007f85842a0000 nid=0x40ab runnable [0x00007f8215291000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000520b941a8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000520b94190> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000520bb30c0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-24" prio=10 tid=0x00007f85d8282800 nid=0x40aa runnable [0x00007f8215392000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000520952288> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000520952270> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000520356888> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-30" prio=10 tid=0x00007f8580328000 nid=0x40a9 runnable [0x00007f8215493000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000520a2fa20> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000520a2fa08> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000520983da0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-23" prio=10 tid=0x00007f85d8267800 nid=0x40a8 runnable [0x00007f8215594000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000052095c470> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000052095c458> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000520356840> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-29" prio=10 tid=0x00007f858030d000 nid=0x40a7 runnable [0x00007f8215695000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000052065b178> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000052065b160> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000520357320> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-28" prio=10 tid=0x00007f85802f2800 nid=0x40a6 runnable [0x00007f8215796000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000052043cad0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000052043cab8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005203567b0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-25" prio=10 tid=0x00007f858829f000 nid=0x40a5 runnable [0x00007f8215897000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000052041a8a8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000052041a890> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000520356690> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-22" prio=10 tid=0x00007f85d824c800 nid=0x40a4 runnable [0x00007f8215998000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000052036b380> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000052036b368> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000520357290> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-24" prio=10 tid=0x00007f8584285800 nid=0x40a3 runnable [0x00007f8215a99000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000052007ac40> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000052007ac28> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051fe41948> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-27" prio=10 tid=0x00007f85802d8000 nid=0x40a2 runnable [0x00007f8215b9a000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005201a2ae8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005201a2ad0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051ffda458> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-21" prio=10 tid=0x00007f85d8232000 nid=0x40a1 waiting for monitor entry [0x00007f8215c9a000] 
> > > > > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > > > > at org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
> > > > > > - waiting to lock <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> > > > > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
> > > > > > at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> > > > > > at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> > > > > > at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> > > > > > at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> > > > > > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > > > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > > > > at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> > > > > > at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> > > > > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> > > > > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > > > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> > > > > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-26" prio=10 tid=0x00007f85802bd800 nid=0x40a0 runnable [0x00007f8215d9c000]
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051ff20e78> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051ff20e60> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051fe41900> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-24" prio=10 tid=0x00007f8588284000 nid=0x409f runnable [0x00007f8215e9d000]
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051fbc1ca0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051fbc1c88> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051f9a7978> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-25" prio=10 tid=0x00007f85802a2800 nid=0x409e runnable [0x00007f8215f9e000]
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051fa3b5b8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051fa3b5a0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051f3155b0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-20" prio=10 tid=0x00007f85d8217800 nid=0x409d runnable [0x00007f821609f000]
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051f83c838> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051f83c820> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051f3150e8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-23" prio=10 tid=0x00007f8588269800 nid=0x409c runnable [0x00007f82161a0000]
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051f78b9b0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051f78b998> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051f3156d0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-22" prio=10 tid=0x00007f858824e800 nid=0x409b runnable [0x00007f82162a1000]
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051f7c0cc8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051f7c0cb0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051f3151c0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-23" prio=10 tid=0x00007f858426a800 nid=0x409a runnable [0x00007f82163a2000]
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051f6065a8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051f606590> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051f3d9668> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-22" prio=10 tid=0x00007f858424f800 nid=0x4099 runnable [0x00007f82164a3000]
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051f56d710> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051f56d6f8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051f314f80> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-21" prio=10 tid=0x00007f8588233800 nid=0x4098 runnable [0x00007f82165a4000]
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051f3bcd68> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051f3bcd50> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051f315400> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-19" prio=10 tid=0x00007f85d81fc800 nid=0x4097 runnable [0x00007f82166a5000]
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051f53b198> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051f53b180> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051f315760> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-20" prio=10 tid=0x00007f8588218800 nid=0x4096 waiting for monitor entry [0x00007f82167a5000]
> > > > > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > > > > at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
> > > > > > - waiting to lock <0x0000000519111590> (a org.apache.flume.channel.file.LogFileV3$Writer)
> > > > > > at org.apache.flume.channel.file.Log.put(Log.java:507)
> > > > > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
> > > > > > at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> > > > > > at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> > > > > > at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> > > > > > at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> > > > > > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > > > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > > > > at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> > > > > > at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> > > > > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> > > > > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > > > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> > > > > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-18" prio=10 tid=0x00007f85d81e1800 nid=0x4095 runnable [0x00007f82168a7000]
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051ee92ba0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051ee92b88> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051ed19740> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-19" prio=10 tid=0x00007f85881fd800 nid=0x4094 runnable [0x00007f82169a8000]
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051ee5da40> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051ee5da28> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051e77a940> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-17" prio=10 tid=0x00007f85d81c6800 nid=0x4093 runnable [0x00007f8216aa9000]
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051ecb1730> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051ecb1718> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051e77a8b0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-21" prio=10 tid=0x00007f8584234800 nid=0x4092 runnable [0x00007f8216baa000]
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051e899660> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051e899648> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051e77a310> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-24" prio=10 tid=0x00007f8580287000 nid=0x4091 runnable [0x00007f8216caa000]
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at org.apache.flume.channel.file.FlumeEventQueue.addTail(FlumeEventQueue.java:165)
> > > > > > - locked <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> > > > > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:529)
> > > > > > - locked <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> > > > > > at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
> > > > > > at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
> > > > > > at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> > > > > > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > > > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > > > > at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> > > > > > at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> > > > > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> > > > > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > > > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> > > > > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-23" prio=10 tid=0x00007f858026c000 nid=0x4090 runnable [0x00007f8216dac000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051ea8b6f0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051ea8b6d8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051e77a820> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-16" prio=10 tid=0x00007f85d81ac000 nid=0x408f runnable [0x00007f8216ead000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051e7c8808> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051e7c87f0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051e77a280> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-20" prio=10 tid=0x00007f8584219800 nid=0x408e runnable [0x00007f8216fae000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051e8fffa8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051e8fff90> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051e77a700> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-22" prio=10 tid=0x00007f8580251000 nid=0x408d runnable [0x00007f82170af000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051e4dc398> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051e4dc380> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051dcbeef8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-19" prio=10 tid=0x00007f85841fe800 nid=0x408c runnable [0x00007f82171b0000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051e221520> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051e221508> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051e407860> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-18" prio=10 tid=0x00007f85841e3800 nid=0x408b runnable [0x00007f82172b1000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051e465298> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051e465280> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051dcbef88> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-18" prio=10 tid=0x00007f85881e3000 nid=0x408a runnable [0x00007f82173b2000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051dea12e0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051dea12c8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051cfd4d78> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-15" prio=10 tid=0x00007f85d8191000 nid=0x4089 runnable [0x00007f82174b3000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051de322b8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051de322a0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051dcbec28> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-14" prio=10 tid=0x00007f85d8176800 nid=0x4088 runnable [0x00007f82175b4000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051dea29a8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051dea2990> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051dcbf018> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-21" prio=10 tid=0x00007f8580236800 nid=0x4087 runnable [0x00007f82176b5000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051dca95d0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051dca95b8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051cfd4e08> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-17" prio=10 tid=0x00007f85841c8800 nid=0x4086 runnable [0x00007f82177b6000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051da8d058> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051da8d040> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051cd693f8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-20" prio=10 tid=0x00007f858021b800 nid=0x4085 runnable [0x00007f82178b7000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051d95a5b0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051d95a598> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051cfd4e98> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-16" prio=10 tid=0x00007f85841ad800 nid=0x4083 runnable [0x00007f82179b8000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051d92a278> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051d92a260> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051cfd4fb8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-13" prio=10 tid=0x00007f85d815b800 nid=0x4082 runnable [0x00007f8217ab9000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051d734788> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051d734770> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051cd69488> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-19" prio=10 tid=0x00007f8580200800 nid=0x4081 runnable [0x00007f8217bba000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051d4268e8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051d4268d0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051d139810> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-15" prio=10 tid=0x00007f8584192800 nid=0x4080 runnable [0x00007f8217cbb000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051d4cda88> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051d4cda70> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051babcf98> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-17" prio=10 tid=0x00007f85881c8000 nid=0x407f runnable [0x00007f8217dbc000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051d32ffe8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051d32ffd0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051cd691b8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-14" prio=10 tid=0x00007f8584177800 nid=0x407e runnable [0x00007f8217ebd000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051d506c60> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051d506c48> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051cfd5048> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-16" prio=10 tid=0x00007f85881ad000 nid=0x407d runnable [0x00007f8217fbe000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051d1223e0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051d1223c8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051c6c7b40> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-15" prio=10 tid=0x00007f8588192800 nid=0x407c runnable [0x00007f82180bf000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051d146f50> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051d146f38> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051cd69248> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-13" prio=10 tid=0x00007f858415c800 nid=0x407b runnable [0x00007f82181c0000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051d11d1b8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051d11d1a0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051c6c7c18> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-12" prio=10 tid=0x00007f85d8141000 nid=0x407a runnable [0x00007f82182c1000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051cd01a78> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051cd01a60> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051cd695a8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-11" prio=10 tid=0x00007f85d8126800 nid=0x4079 runnable [0x00007f82183c2000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051cfa6f88> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051cfa6f70> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051ba67370> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-14" prio=10 tid=0x00007f8588177800 nid=0x4078 runnable [0x00007f82184c3000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051ce5cdd0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051ce5cdb8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051c6c7bd0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-12" prio=10 tid=0x00007f8584142000 nid=0x4077 runnable [0x00007f82185c4000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051cc27cb8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051cc27ca0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051babd0b8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-18" prio=10 tid=0x00007f85801e6000 nid=0x4076 runnable [0x00007f82186c5000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051cb895c8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051cc77fd0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051c6c7cf0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-11" prio=10 tid=0x00007f8584127800 nid=0x4075 runnable [0x00007f82187c6000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051ca540d0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051caad368> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051c6c7e10> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-13" prio=10 tid=0x00007f858815c800 nid=0x4074 runnable [0x00007f82188c7000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051c953ab8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051ca799a8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051babd028> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-17" prio=10 tid=0x00007f85801cb800 nid=0x4073 runnable [0x00007f82189c8000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051c77c3d8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051c8137c8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051c3856f0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-10" prio=10 tid=0x00007f858410c800 nid=0x4072 runnable [0x00007f8218ac9000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051c7dd2b0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051c613988> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051babcde8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-16" prio=10 tid=0x00007f85801b1000 nid=0x4071 runnable [0x00007f8218bca000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051c776580> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051c6633c8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051c6c7ea0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-9" prio=10 tid=0x00007f85840f1800 nid=0x4070 runnable [0x00007f8218ccb000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051c36b550> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051c36b568> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051ba675f8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-8" prio=10 tid=0x00007f85840d6800 nid=0x406f runnable [0x00007f8218dcc000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051c546b00> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051c40a178> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051ba67760> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-7" prio=10 tid=0x00007f85840bb800 nid=0x406e runnable [0x00007f8218ecd000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051c2ecf90> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051c29a100> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051ba675b0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-15" prio=10 tid=0x00007f8580196000 nid=0x406d runnable [0x00007f8218fce000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051c2ee670> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051c2ee688> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051ba67718> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-12" prio=10 tid=0x00007f8588141800 nid=0x406c runnable [0x00007f82190cf000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051c0bd978> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051c16cc00> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051babcf08> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-6" prio=10 tid=0x00007f85840a0800 nid=0x406b runnable [0x00007f82191d0000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051c068508> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051bf825f0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051ba676d0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-11" prio=10 tid=0x00007f8588127000 nid=0x406a runnable [0x00007f82192d1000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051becaa68> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051c03ead0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051babce78> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-10" prio=10 tid=0x00007f858810c800 nid=0x4069 runnable [0x00007f82193d2000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051be31b98> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051bd56790> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051ba67490> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-14" prio=10 tid=0x00007f858017b000 nid=0x4068 runnable [0x00007f82194d3000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051bccb8c0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051bb14420> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051babcd58> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-5" prio=10 tid=0x00007f8584085800 nid=0x4067 runnable [0x00007f82195d4000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051bd004a8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051bd49480> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051ba67400> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-4" prio=10 tid=0x00007f858406b000 nid=0x4066 runnable [0x00007f82196d5000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051ba1e928> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051bccdd00> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051ba66ef0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-9" prio=10 tid=0x00007f85880f1800 nid=0x4065 waiting for monitor entry [0x00007f82197d5000] 
> > > > > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > > > > at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
> > > > > > - waiting to lock <0x0000000519111590> (a org.apache.flume.channel.file.LogFileV3$Writer)
> > > > > > at org.apache.flume.channel.file.Log.put(Log.java:507)
> > > > > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
> > > > > > at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> > > > > > at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> > > > > > at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> > > > > > at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> > > > > > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > > > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > > > > at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> > > > > > at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> > > > > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> > > > > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > > > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> > > > > > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> > > > > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > > > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > > > > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-8" prio=10 tid=0x00007f85880d6800 nid=0x4064 runnable [0x00007f82198d7000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051b5bcca8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051b7ef8f0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051b5c7cb8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-7" prio=10 tid=0x00007f85880bb800 nid=0x4063 runnable [0x00007f82199d8000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051b788ce8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051b788d00> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051b285260> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-13" prio=10 tid=0x00007f8580160800 nid=0x4062 runnable [0x00007f8219ad9000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051b71a0f8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051b750e98> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051aed5a70> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-10" prio=10 tid=0x00007f85d810b800 nid=0x4061 runnable [0x00007f8219bda000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051b517878> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051b5bab30> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051aed5b90> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-9" prio=10 tid=0x00007f85d80f0800 nid=0x4060 runnable [0x00007f8219cdb000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051b5656e0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051b5657a0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051b285338> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-8" prio=10 tid=0x00007f85d80d5800 nid=0x405f runnable [0x00007f8219ddc000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051b3df220> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051b475118> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051aed5c20> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-6" prio=10 tid=0x00007f85880a0800 nid=0x405e runnable [0x00007f8219edd000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051b11c010> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051b1eab78> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051af91478> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-7" prio=10 tid=0x00007f85d80bb000 nid=0x405d runnable [0x00007f8219fde000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051b1d36e0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051b0328c0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051aed5cb0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-12" prio=10 tid=0x00007f8580145800 nid=0x405c runnable [0x00007f821a0df000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051aeb3070> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051aeb3088> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051a98a820> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-6" prio=10 tid=0x00007f85d80a0000 nid=0x405b runnable [0x00007f821a1e0000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051aefa920> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051af3ee40> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051ac28810> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-5" prio=10 tid=0x00007f8588086000 nid=0x405a runnable [0x00007f821a2e1000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051aefe1b0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051ad2b858> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051aed5e18> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-11" prio=10 tid=0x00007f858012a800 nid=0x4059 runnable [0x00007f821a3e2000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051ac42588> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051ad05d70> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051a98a7d8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-3" prio=10 tid=0x00007f8584050000 nid=0x4058 runnable [0x00007f821a4e3000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051ad06100> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051ad47428> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051ac289c0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-10" prio=10 tid=0x00007f858010f800 nid=0x4057 runnable [0x00007f821a5e4000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051aab5d80> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051ab8ce38> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051a98a748> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-5" prio=10 tid=0x00007f85d8085800 nid=0x4056 runnable [0x00007f821a6e5000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051a962980> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051a775fb0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051a842098> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-2" prio=10 tid=0x00007f8584035000 nid=0x4055 runnable [0x00007f821a7e6000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051a817210> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051a7d2b48> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051a841ca8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-4" prio=10 tid=0x00007f85d806b000 nid=0x4054 runnable [0x00007f821a8e7000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051a903340> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051a903358> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051a42db50> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-9" prio=10 tid=0x00007f85800f4800 nid=0x4053 runnable [0x00007f821a9e8000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051a4f17d8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051a37ae08> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051a22e340> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-4" prio=10 tid=0x00007f858806b000 nid=0x4052 runnable [0x00007f821aae9000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051a4d81e8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051a616bb0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051a1dfb40> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-3" prio=10 tid=0x00007f85d8050000 nid=0x4051 runnable [0x00007f821abea000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051a4fb240> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051a3913b8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051a213878> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-2" prio=10 tid=0x00007f85d8035800 nid=0x4050 runnable [0x00007f821aceb000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051a506248> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051a506308> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051a42dd48> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-8" prio=10 tid=0x00007f85800d9800 nid=0x404f runnable [0x00007f821adec000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051a2d5170> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051a2d5188> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051a22e2b0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-7" prio=10 tid=0x00007f85800be800 nid=0x404e runnable [0x00007f821aeed000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051a2d3cb8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051a2b8310> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051a213758> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10 tid=0x00007f857b378800 nid=0x404d waiting for monitor entry [0x00007f821afee000] 
> > > > > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > > > > at org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
> > > > > > - waiting to lock <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> > > > > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
> > > > > > at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> > > > > > at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> > > > > > at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
> > > > > > at org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> > > > > > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f836c01a000 nid=0x404c runnable [0x00007f821b0ef000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051a045bb0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051a045bc8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051a000c98> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b376800 nid=0x404b waiting on condition [0x00007f821b1f0000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051a045c10> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10 tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.NativeThread.current(Native Method)
> > > > > > at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
> > > > > > at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
> > > > > > - locked <0x00000005190ec998> (a java.lang.Object)
> > > > > > at org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
> > > > > > at org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
> > > > > > - locked <0x0000000519111590> (a org.apache.flume.channel.file.LogFileV3$Writer)
> > > > > > at org.apache.flume.channel.file.Log.take(Log.java:550)
> > > > > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
> > > > > > at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> > > > > > at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> > > > > > at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
> > > > > > at org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> > > > > > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8354035000 nid=0x4049 runnable [0x00007f821b3f2000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051a045c28> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051a045c40> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051a000ce0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834801a000 nid=0x4048 runnable [0x00007f821b4f3000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051a17bd60> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051a17bd78> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051a180230> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b374800 nid=0x4047 waiting on condition [0x00007f821b5f4000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051a08ba88> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857338a800 nid=0x4046 waiting on condition [0x00007f821b6f5000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051a147940> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10 tid=0x00007f85f3bff800 nid=0x4045 waiting on condition [0x00007f821b7f6000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000056dca65c0> (a java.util.concurrent.FutureTask$Sync)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
> > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
> > > > > > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:227)
> > > > > > at java.util.concurrent.FutureTask.get(FutureTask.java:91)
> > > > > > at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:323)
> > > > > > at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:269)
> > > > > > at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:352)
> > > > > > at org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> > > > > > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8344035000 nid=0x4044 runnable [0x00007f821b8f7000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051a184a90> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051a184aa8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051a0bb2e0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bfd800 nid=0x4043 waiting on condition [0x00007f821b9f8000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051a147958> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f833001a000 nid=0x4042 runnable [0x00007f821baf9000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051a188f28> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051a188f40> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519d90758> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8334035000 nid=0x4041 runnable [0x00007f821bbfa000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051a045cd0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051a045ce8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051a000d28> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b372800 nid=0x4040 waiting on condition [0x00007f821bcfb000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519f74590> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573388800 nid=0x403f waiting on condition [0x00007f821bdfc000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519fb0c90> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832c035000 nid=0x403e runnable [0x00007f821befd000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051a045d30> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051a045d48> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051a000d70> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bfb800 nid=0x403d waiting on condition [0x00007f821bffe000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519e5d388> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f831801a000 nid=0x403c runnable [0x00007f82341c0000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519ebdb70> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519ebdb88> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519d3a778> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f831001a000 nid=0x403b runnable [0x00007f82342c1000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519f47f90> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519f47fa8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519e11ca8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaf0000 nid=0x403a waiting on condition [0x00007f82343c2000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519f73018> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8308035000 nid=0x4039 runnable [0x00007f82344c3000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519f745a8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519f745c0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519ca9f78> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573386800 nid=0x4038 waiting on condition [0x00007f82345c4000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519fa5ce0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bf9800 nid=0x4037 waiting on condition [0x00007f82346c5000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519fb0ca8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8300035000 nid=0x4036 runnable [0x00007f82347c6000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519f22d20> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519f22d38> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519d9f4b8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaee000 nid=0x4035 waiting on condition [0x00007f82348c7000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519fb0cc0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82fc01a000 nid=0x4034 runnable [0x00007f82349c8000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519f74608> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519f74620> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519ca9fc0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82f001a000 nid=0x4033 runnable [0x00007f8234ac9000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519fb0cd8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519fb0cf0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519cadd40> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f44800 nid=0x4032 waiting on condition [0x00007f8234bca000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519ebdbd0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3377000 nid=0x4031 waiting on condition [0x00007f8234ccb000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519f701c0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82e8035000 nid=0x4030 runnable [0x00007f8234dcc000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519f47ff0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519f48008> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519e11cf0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaec000 nid=0x402f waiting on condition [0x00007f8234ecd000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519fa5cf8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10 tid=0x00007f857fc38000 nid=0x402e waiting for monitor entry [0x00007f8234fce000] 
> > > > > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > > > > at org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
> > > > > > - waiting to lock <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> > > > > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
> > > > > > at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> > > > > > at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> > > > > > at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
> > > > > > at org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> > > > > > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82dc01a000 nid=0x402d runnable [0x00007f82350cf000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519ebdbe8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519ebdc00> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519d3a7c0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82d001a000 nid=0x402c runnable [0x00007f82351d0000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519fb0d38> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519fb0d50> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519cadd88> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-144-thread-1" prio=10 tid=0x00007f85d8001000 nid=0x402b runnable [0x00007f82352d1000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519f481a0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519f49040> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519e11d80> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6d3800 nid=0x402a waiting on condition [0x00007f82353d2000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519f22d80> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f42800 nid=0x4029 waiting on condition [0x00007f82354d3000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519fa5d10> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82cc035000 nid=0x4028 runnable [0x00007f82355d4000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519fb0d98> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519fb0db0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519caddd0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-3" prio=10 tid=0x00007f8588001000 nid=0x4027 runnable [0x00007f82356d5000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519ebdc48> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519ebdc60> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519d3a808> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3375000 nid=0x4026 waiting on condition [0x00007f82357d6000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519fa5d28> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82c4035000 nid=0x4025 runnable [0x00007f82358d7000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519f22d98> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519f22db0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519d9f500> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaea000 nid=0x4024 waiting on condition [0x00007f82359d8000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519fa5d40> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a8035000 nid=0x4023 runnable [0x00007f8235ad9000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519fb0df8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519fb0e10> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519cade18> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82b401a000 nid=0x4022 runnable [0x00007f8235bda000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519f73030> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519f73048> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519e69de8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82ac035000 nid=0x4021 runnable [0x00007f8235cdb000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519f61380> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519f61398> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005197600f0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6d1800 nid=0x4020 waiting on condition [0x00007f8235ddc000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519f22df8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f40800 nid=0x401f waiting on condition [0x00007f8235edd000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519fa5d58> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3373000 nid=0x401e waiting on condition [0x00007f8235fde000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519ebdca8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a4035000 nid=0x401d runnable [0x00007f82360df000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519fb0e58> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519fb0e70> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519cade60> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a8000 nid=0x401c waiting on condition [0x00007f82361e0000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519f22e10> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-6" prio=10 tid=0x00007f85800a3800 nid=0x401b runnable [0x00007f82362e1000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519ebde28> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519de7350> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519d3a898> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828801a000 nid=0x401a runnable [0x00007f82363e2000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519fa5d70> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519fa5d88> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519d36338> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828c035000 nid=0x4019 runnable [0x00007f82364e3000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519f73090> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519f730a8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519e69e30> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8280035000 nid=0x4018 runnable [0x00007f82365e4000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519fb0eb8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519e1a7f8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519cadea8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6cf800 nid=0x4017 waiting on condition [0x00007f82366e5000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519f22e28> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f3e800 nid=0x4016 waiting on condition [0x00007f82367e6000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519f701d8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3371000 nid=0x4015 waiting on condition [0x00007f82368e7000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519fb0f00> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8278035000 nid=0x4014 runnable [0x00007f82369e8000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519e93628> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519e93640> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519d9f548> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a6000 nid=0x4013 waiting on condition [0x00007f8236ae9000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005199bdaf0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f827001a000 nid=0x4012 runnable [0x00007f8236bea000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519fb0f18> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519fb0f30> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519cadef0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6cd800 nid=0x4011 waiting on condition [0x00007f8236ceb000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519c47f38> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826c035000 nid=0x4010 runnable [0x00007f8236dec000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519fa5dd0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519fa5de8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519d36380> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826001a000 nid=0x400f runnable [0x00007f8236eed000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519c8c170> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519c8c188> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519a73d78> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572aeb000 nid=0x400e waiting on condition [0x00007f8236fee000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519c7e570> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2e800 nid=0x400d waiting on condition [0x00007f82370ef000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519c47f50> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8258035000 nid=0x400b runnable [0x00007f82371f0000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519c4b190> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519c4b1a8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005199eb068> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a4000 nid=0x400a waiting on condition [0x00007f82372f1000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519c47f68> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f825001a000 nid=0x4009 runnable [0x00007f82373f2000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519c33550> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519c33568> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051996b7b8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e29c000 nid=0x4008 waiting on condition [0x00007f82374f3000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519c47f80> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f824c03f800 nid=0x4007 runnable [0x00007f82375f4000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519c7e588> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519c7e5a0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005199a1910> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8240035000 nid=0x4006 runnable [0x00007f82376f5000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519c47f98> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519c47fb0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005199536b8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae9000 nid=0x4005 waiting on condition [0x00007f82377f6000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519c368c8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2c800 nid=0x4004 waiting on condition [0x00007f82378f7000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519c29bf0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8238035000 nid=0x4003 runnable [0x00007f82379f8000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519bd8198> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519bd81b0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519b59da0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a2000 nid=0x4002 waiting on condition [0x00007f8237af9000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519c7e5e8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8230035000 nid=0x4001 runnable [0x00007f8237bfa000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519c47ff8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519c48010> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519953700> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e29a000 nid=0x4000 waiting on condition [0x00007f8237cfb000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519c7e600> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f823001a000 nid=0x3fff runnable [0x00007f8237dfc000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519c7e618> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519c7e630> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005199a1958> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f823c01a000 nid=0x3ffe runnable [0x00007f8237efd000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519c48058> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519c48070> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519a6d798> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f823801a000 nid=0x3ffd runnable [0x00007f8237ffe000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519c335b0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519c335c8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051996b800> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae7000 nid=0x3ffc waiting on condition [0x00007f84001c0000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519c480b8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2a800 nid=0x3ffb waiting on condition [0x00007f84002c1000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519bd81f8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24e800 nid=0x3ffa waiting on condition [0x00007f84003c2000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519c29c08> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f824001a000 nid=0x3ff9 runnable [0x00007f84004c3000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519c7e678> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519c7e690> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005199a19a0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e298000 nid=0x3ff8 waiting on condition [0x00007f84005c4000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519c67408> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-5" prio=10 tid=0x00007f8580088800 nid=0x3ff7 runnable [0x00007f84006c5000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519c48238> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519d5e0d0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519a6d828> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f825c01a000 nid=0x3ff6 runnable [0x00007f84007c6000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519c368e0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519c368f8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519953088> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f825801a000 nid=0x3ff5 runnable [0x00007f84008c7000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519c33610> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519c33628> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051996b848> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826401a000 nid=0x3ff4 runnable [0x00007f84009c8000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519c7e6d8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519c7e6f0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005199a19e8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae5000 nid=0x3ff3 waiting on condition [0x00007f8400ac9000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519c730b8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24c800 nid=0x3ff2 waiting on condition [0x00007f8400bca000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519c29c20> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f29000 nid=0x3ff1 waiting on condition [0x00007f8400ccb000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519c4b1f0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826c01a000 nid=0x3ff0 runnable [0x00007f8400dcc000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519bd8210> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519bd8228> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519b59de8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e296800 nid=0x3fef waiting on condition [0x00007f8400ecd000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519c8c1d0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f827801a000 nid=0x3fee runnable [0x00007f8400fce000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519c7e738> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519c7e750> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005199a1a30> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828401a000 nid=0x3fed runnable [0x00007f84010cf000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519c33670> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519c33688> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051996b890> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828001a000 nid=0x3fec runnable [0x00007f84011d0000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519c29c38> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519c29c50> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519955c68> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828c01a000 nid=0x3feb runnable [0x00007f84012d1000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519c7e798> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519c7e7b0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005199533a0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae6000 nid=0x3fea waiting on condition [0x00007f84013d2000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519bd8270> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24a800 nid=0x3fe9 waiting on condition [0x00007f84014d3000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519c36940> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572273800 nid=0x3fe8 waiting on condition [0x00007f84015d4000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519c8c1e8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de54800 nid=0x3fe7 waiting on condition [0x00007f84016d5000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519c67420> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f829001a000 nid=0x3fe6 runnable [0x00007f84017d6000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519c7e7f8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519c7e810> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005199533e8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a401a000 nid=0x3fe5 runnable [0x00007f84018d7000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519c336d0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519c336e8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051996b8d8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae4000 nid=0x3fe4 waiting on condition [0x00007f84019d8000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519946350> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-4" prio=10 tid=0x00007f858006d800 nid=0x3fe3 runnable [0x00007f8401ad9000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519bd83d8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519f042a0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519b59e78> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82ac01a000 nid=0x3fe2 runnable [0x00007f8401bda000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519c7e858> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519c7e870> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519953430> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a248800 nid=0x3fe1 waiting on condition [0x00007f8401cdb000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519890ec8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a801a000 nid=0x3fe0 runnable [0x00007f8401ddc000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519c8c200> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519c8c218> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519a73dc0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572271800 nid=0x3fdf waiting on condition [0x00007f8401edd000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005198a3170> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de52800 nid=0x3fde waiting on condition [0x00007f8401fde000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005198b7610> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82b801a000 nid=0x3fdd runnable [0x00007f84020df000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519947f70> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519947f88> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005197bc938> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82c401a000 nid=0x3fdc runnable [0x00007f84021e0000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519946368> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519946380> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051983b9b8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82cc01a000 nid=0x3fdb runnable [0x00007f84022e1000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005198a3188> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005198a31a0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005197300e0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae2000 nid=0x3fda waiting on condition [0x00007f84023e2000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005199463c8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857226f800 nid=0x3fd9 waiting on condition [0x00007f84024e3000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519792380> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82c801a000 nid=0x3fd8 runnable [0x00007f84025e4000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519947fd0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519947fe8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005197bc980> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579597000 nid=0x3fd7 waiting on condition [0x00007f84026e5000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519488cc0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de50800 nid=0x3fd6 waiting on condition [0x00007f84027e6000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005198cc640> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82d801a000 nid=0x3fd5 runnable [0x00007f84028e7000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005199463e0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005199463f8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051983ba00> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82e401a000 nid=0x3fd4 runnable [0x00007f84029e8000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005198f7fc8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005198f7fe0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005194d0160> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82ec01a000 nid=0x3fd3 runnable [0x00007f8402ae9000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005198a31e8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005198a3200> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519730128> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae0800 nid=0x3fd2 waiting on condition [0x00007f8402bea000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519948030> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857226d800 nid=0x3fd1 waiting on condition [0x00007f8402ceb000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519946440> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82e801a000 nid=0x3fd0 runnable [0x00007f8402dec000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519890ee0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519890ef8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005196f56f8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579595000 nid=0x3fcf waiting on condition [0x00007f8402eed000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005198b7628> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de4e800 nid=0x3fce waiting on condition [0x00007f8402fee000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005198a3248> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830401a000 nid=0x3fcd runnable [0x00007f84030ef000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519946458> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519946470> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051983ba48> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830001a000 nid=0x3fcc runnable [0x00007f84031f0000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519937b88> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519937ba0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519887cb8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830c01a000 nid=0x3fcb runnable [0x00007f84032f1000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519948048> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519948060> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005197bc9c8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830801a000 nid=0x3fca runnable [0x00007f84033f2000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005199464b8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005199464d0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051983ba90> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e2b000 nid=0x3fc9 waiting on condition [0x00007f84034f3000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519943980> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1595000 nid=0x3fc8 waiting on condition [0x00007f84035f4000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005198a3260> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857bff5800 nid=0x3fc7 waiting on condition [0x00007f84036f5000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519901138> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da0c000 nid=0x3fc6 waiting on condition [0x00007f84037f6000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005199480a8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832001a000 nid=0x3fc5 runnable [0x00007f84038f7000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519946518> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519946530> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051983bad8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832c01a000 nid=0x3fc4 runnable [0x00007f84039f8000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005198a3278> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005198a3290> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519730170> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832801a000 nid=0x3fc3 runnable [0x00007f8403af9000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005198f8028> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005198f8040> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005194d01a8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f833401a000 nid=0x3fc2 runnable [0x00007f8403bfa000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005198b7640> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005198b7658> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005196f3480> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e29000 nid=0x3fc1 waiting on condition [0x00007f8403cfb000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519946578> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1593000 nid=0x3fc0 waiting on condition [0x00007f8403dfc000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005199480c0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579592000 nid=0x3fbf waiting on condition [0x00007f8403efd000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519943998> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da0a800 nid=0x3fbe waiting on condition [0x00007f84401c0000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519946590> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834401a000 nid=0x3fbd runnable [0x00007f84402c1000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519903410> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519903428> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005191a1498> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834001a000 nid=0x3fbc runnable [0x00007f84403c2000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519943ef0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519943f08> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005196f5740> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834c01a000 nid=0x3fbb runnable [0x00007f84404c3000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005199497b0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005199497c8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519685f60> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573ff1800 nid=0x3fba waiting on condition [0x00007f84405c4000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005198f8088> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1591800 nid=0x3fb9 waiting on condition [0x00007f84406c5000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005198a32d8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579591000 nid=0x3fb8 waiting on condition [0x00007f84407c6000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005199480d8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f835401a000 nid=0x3fb7 runnable [0x00007f84408c7000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005199465a8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005199465c0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051983bb20> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da08800 nid=0x3fb6 waiting on condition [0x00007f84409c8000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005199439b0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-142-thread-1" prio=10 tid=0x00007f8584001000 nid=0x3fb5 runnable [0x00007f8440ac9000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005198b77f0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519a00960> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519952e48> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f837401a000 nid=0x3fb4 runnable [0x00007f8440bca000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519937be8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519937c00> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519887d00> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f836801a000 nid=0x3fb3 runnable [0x00007f8440ccb000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519943f50> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519943f68> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005196f5788> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f837001a000 nid=0x3fb2 runnable [0x00007f8440dcc000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519901150> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519901168> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519106100> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f158f800 nid=0x3fb1 waiting on condition [0x00007f8440ecd000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519946608> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e26800 nid=0x3fb0 waiting on condition [0x00007f8440fce000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005198a32f0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913e000 nid=0x3faf waiting on condition [0x00007f84410cf000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005199439c8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f837801a000 nid=0x3fae runnable [0x00007f84411d0000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005199480f0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519948108> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005197bca10> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da07000 nid=0x3fad waiting on condition [0x00007f84412d1000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519943fb0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f838801a000 nid=0x3fac runnable [0x00007f84413d2000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005198f80a0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005198f80b8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005194d01f0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f839401a000 nid=0x3fab runnable [0x00007f84414d3000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519946620> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519946638> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051983bb68> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f839001a000 nid=0x3faa runnable [0x00007f84415d4000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519937c48> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519937c60> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519887d48> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d0d000 nid=0x3fa9 waiting on condition [0x00007f84416d5000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519654310> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913c000 nid=0x3fa8 waiting on condition [0x00007f84417d6000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005191b54d8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571164000 nid=0x3fa7 waiting on condition [0x00007f84418d7000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519637a20> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83a801a000 nid=0x3fa6 runnable [0x00007f84419d8000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005195f76c8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005196430c0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005194ecaf0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83b401a000 nid=0x3fa5 runnable [0x00007f8441ad9000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051965f960> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051965f978> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519104648> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83b001a000 nid=0x3fa4 runnable [0x00007f8441bda000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519655e28> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519655e40> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519104ba0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-3" prio=10 tid=0x00007f8580052800 nid=0x3fa3 runnable [0x00007f8441cdb000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519637b10> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519694210> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051916e518> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83bc01a000 nid=0x3fa2 runnable [0x00007f8441ddc000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519597678> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519597690> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519106388> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c4800 nid=0x3fa1 waiting on condition [0x00007f8441edd000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519654328> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d0b000 nid=0x3fa0 waiting on condition [0x00007f8441fde000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519676ad0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571162000 nid=0x3f9f waiting on condition [0x00007f84420df000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519655618> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913a000 nid=0x3f9e waiting on condition [0x00007f84421e0000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005194ae248> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-2" prio=10 tid=0x00007f8580001000 nid=0x3f9d runnable [0x00007f84422e1000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519654340> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519654358> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519105380> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83cc01a000 nid=0x3f9c runnable [0x00007f84423e2000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519655e88> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519655ea0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519104be8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83d001a000 nid=0x3f9b runnable [0x00007f84424e3000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005194e1350> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005194e1368> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005191041c8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c3000 nid=0x3f9a waiting on condition [0x00007f84425e4000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051967caf8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83d801a000 nid=0x3f99 runnable [0x00007f84426e5000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519655ee8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519655f00> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519104c30> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83dc01a000 nid=0x3f98 runnable [0x00007f84427e6000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005196543a0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005196543b8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005191053c8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d09800 nid=0x3f97 waiting on condition [0x00007f84428e7000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519655630> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579138000 nid=0x3f96 waiting on condition [0x00007f84429e8000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005196592c8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571160000 nid=0x3f95 waiting on condition [0x00007f8442ae9000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519655f48> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f840801a000 nid=0x3f94 runnable [0x00007f8442bea000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519655648> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519655660> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051914b290> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83e801a000 nid=0x3f93 runnable [0x00007f8442ceb000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519643108> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519643120> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005194ecb38> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d07800 nid=0x3f92 waiting on condition [0x00007f8442dec000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519658428> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83f801a000 nid=0x3f91 runnable [0x00007f8442eed000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005194c9908> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005194c9920> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519511968> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83fc01a000 nid=0x3f90 runnable [0x00007f8442fee000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519643168> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519643180> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005194ecb80> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578465000 nid=0x3f8f waiting on condition [0x00007f8403ffe000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519654400> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857115e800 nid=0x3f8e waiting on condition [0x00007f84430ef000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005196556a8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c1000 nid=0x3f8d waiting on condition [0x00007f84431f0000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519654418> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f840c01a000 nid=0x3f8c runnable [0x00007f84432f1000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005196431c8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005196431e0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005194ecbc8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f841801a000 nid=0x3f8b runnable [0x00007f84433f2000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519655f60> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519655f78> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519104c78> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b4000 nid=0x3f8a waiting on condition [0x00007f84434f3000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005196721e8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5bf000 nid=0x3f89 waiting on condition [0x00007f84435f4000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519654430> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f841c01a000 nid=0x3f88 runnable [0x00007f84436f5000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005196556c0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005196556d8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051914b2d8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f842801a000 nid=0x3f87 runnable [0x00007f84437f6000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519643228> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519643240> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005194ecc10> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b2000 nid=0x3f86 waiting on condition [0x00007f84438f7000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519654448> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5bd800 nid=0x3f85 waiting on condition [0x00007f84439f8000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051967cb10> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f843001a000 nid=0x3f84 runnable [0x00007f8443af9000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519655fc0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519655fd8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519104cc0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b0800 nid=0x3f83 waiting on condition [0x00007f8443bfa000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005196592e0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f843801a000 nid=0x3f82 runnable [0x00007f8443cfb000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519643288> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005196432a0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005194ecc58> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d17b000 nid=0x3f81 waiting on condition [0x00007f8443dfc000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051921afd8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f843c01a000 nid=0x3f80 runnable [0x00007f8443efd000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519654460> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519654478> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519105410> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08ae800 nid=0x3f7f waiting on condition [0x00007f8443ffe000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519676ae8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f844401a000 nid=0x3f7e runnable [0x00007f85441a6000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051967cb28> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051967cb40> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005194745e0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d179000 nid=0x3f7d waiting on condition [0x00007f85442a7000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005196432e8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f845801a000 nid=0x3f7c runnable [0x00007f85443a8000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519658440> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005196583f8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519349128> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f845401a000 nid=0x3f7b runnable [0x00007f85444a9000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519676aa0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519676ab8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051921b020> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08ad800 nid=0x3f7a waiting on condition [0x00007f85446c5000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519659298> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578463000 nid=0x3f79 waiting on condition [0x00007f85447c6000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005194a3568> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f845c01a000 nid=0x3f78 runnable [0x00007f85445aa000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005196544c0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005196544d8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519105458> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d177000 nid=0x3f77 waiting on condition [0x00007f85448c7000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519656020> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f846401a000 nid=0x3f76 runnable [0x00007f85449c8000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519655720> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519655738> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519105f98> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857115c800 nid=0x3f75 waiting on condition [0x00007f8544ac9000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519402f58> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f847801a000 nid=0x3f74 runnable [0x00007f8544bca000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519654520> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519654538> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005191054a0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f847401a000 nid=0x3f73 runnable [0x00007f8544ccb000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at java.util.HashMap.newKeyIterator(HashMap.java:840)
> > > > > > at java.util.HashMap$KeySet.iterator(HashMap.java:874)
> > > > > > at java.util.HashSet.iterator(HashSet.java:153)
> > > > > > at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
> > > > > > - locked <0x0000000519656248> (a java.util.HashSet)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519656038> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519656050> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519104d08> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0015800 nid=0x3f72 waiting on condition [0x00007f8544dcc000]
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051941d010> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578461000 nid=0x3f71 waiting on condition [0x00007f8544ecd000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519477938> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f847c01a000 nid=0x3f70 runnable [0x00007f8544fce000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051942aa08> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051942aa20> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519349050> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d175800 nid=0x3f6f waiting on condition [0x00007f85450cf000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519470c60> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f848401a000 nid=0x3f6e runnable [0x00007f85451d0000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519445fa0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519445fb8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005194679d0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570456800 nid=0x3f6d waiting on condition [0x00007f85452d1000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519419230> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f849801a000 nid=0x3f6c runnable [0x00007f85453d2000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519470c78> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519470c90> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005194743a0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f849401a000 nid=0x3f6b runnable [0x00007f85454d3000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051945beb8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051945bed0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519196000> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0013800 nid=0x3f6a waiting on condition [0x00007f85455d4000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519470cd8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857845f000 nid=0x3f69 waiting on condition [0x00007f85456d5000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051941d1c0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f849c01a000 nid=0x3f68 runnable [0x00007f85457d6000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005194583b8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005194583d0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005194743e8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c488000 nid=0x3f67 waiting on condition [0x00007f85458d7000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519424e80> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84a401a000 nid=0x3f66 runnable [0x00007f85459d8000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051941d1d8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051941d1f0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519467a18> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570455000 nid=0x3f65 waiting on condition [0x00007f8545ad9000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051945bf18> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84b801a000 nid=0x3f64 runnable [0x00007f8545bda000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519419248> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519419200> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519431ee0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84c001a000 nid=0x3f63 runnable [0x00007f8545cdb000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051941d238> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051941d250> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519467a60> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0012000 nid=0x3f62 waiting on condition [0x00007f8545ddc000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519458418> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84bc01a000 nid=0x3f61 runnable [0x00007f8545edd000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051945bf30> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051945bf48> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519196048> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857845e000 nid=0x3f60 waiting on condition [0x00007f8545fde000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519479f10> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c486000 nid=0x3f5f waiting on condition [0x00007f85460df000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051941d298> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84c401a000 nid=0x3f5e runnable [0x00007f85461e0000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051942a9a8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051942a9c0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519349098> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570453800 nid=0x3f5d waiting on condition [0x00007f85462e1000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051945bf90> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84cc01a000 nid=0x3f5c runnable [0x00007f85463e2000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519458430> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519458448> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519474430> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c484800 nid=0x3f5b waiting on condition [0x00007f85464e3000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051941d2b0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84d401a000 nid=0x3f5a runnable [0x00007f85465e4000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005194191e8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051936a4e0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519431f28> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570450800 nid=0x3f59 waiting on condition [0x00007f85466e5000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051943ef60> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84dc01a000 nid=0x3f58 runnable [0x00007f85467e6000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051942a948> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051942a960> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005193490e0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c482800 nid=0x3f57 waiting on condition [0x00007f85468e7000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051945bfa8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84e401a000 nid=0x3f56 runnable [0x00007f85469e8000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051941d2c8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051941d2e0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519467aa8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857044f000 nid=0x3f55 waiting on condition [0x00007f8546ae9000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051945bfc0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84f801a000 nid=0x3f54 runnable [0x00007f8546bea000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051936a4c8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051936a480> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519431f70> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84f401a000 nid=0x3f53 runnable [0x00007f8546ceb000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519458490> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005194584a8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519474478> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578008000 nid=0x3f52 waiting on condition [0x00007f8546dec000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051942a8e8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c00d000 nid=0x3f51 waiting on condition [0x00007f8546eed000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519447098> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84fc01a000 nid=0x3f50 runnable [0x00007f8546fee000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051941d328> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051941d340> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519467af0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857044d000 nid=0x3f4f waiting on condition [0x00007f85470ef000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051945bfd8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f851001a000 nid=0x3f4e runnable [0x00007f85471f0000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519424e98> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519424e50> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519380d20> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f850c01a000 nid=0x3f4d runnable [0x00007f85472f1000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005194584f0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519458508> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005194744c0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578006800 nid=0x3f4c waiting on condition [0x00007f85473f2000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051945bff0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c01d800 nid=0x3f4b waiting on condition [0x00007f85474f3000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051936a468> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f851401a000 nid=0x3f4a runnable [0x00007f85475f4000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051941d388> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051941d3a0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519467b38> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857045b800 nid=0x3f49 waiting on condition [0x00007f85476f5000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051942a900> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f853801a000 nid=0x3f48 runnable [0x00007f85477f6000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519479f28> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519479ee0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519203068> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f853001f800 nid=0x3f47 runnable [0x00007f85478f7000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519477950> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519477968> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051922faf0> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f852c01f000 nid=0x3f46 runnable [0x00007f85479f8000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005194353f8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519435410> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x000000051921b0f8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f853401a000 nid=0x3f45 runnable [0x00007f8547af9000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051945c008> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051945c020> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000519196090> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578005800 nid=0x3f44 waiting on condition [0x00007f8547bfa000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005190de788> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f001a800 nid=0x3f43 waiting on condition [0x00007f8547cfb000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000518fae040> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c022000 nid=0x3f42 waiting on condition [0x00007f8547dfc000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000518cafb20> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570460000 nid=0x3f41 waiting on condition [0x00007f8547efd000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005190db9c8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > > > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > > > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-2" prio=10 tid=0x00007f8588036000 nid=0x3f40 runnable [0x00007f8547ffe000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005190ee0e8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005191f0be8> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005190e1d18> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-140-thread-1" prio=10 tid=0x00007f858801b800 nid=0x3f3f runnable [0x00007f858c18a000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000518f16658> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051941f820> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005190edc80> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-139-thread-1" prio=10 tid=0x00007f8580002800 nid=0x3f3e runnable [0x00007f858c28b000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x000000051913eb50> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x000000051913eca0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000518c9c4e8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > > > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > > > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-148-thread-1" prio=10 tid=0x00007f856844c000 nid=0x3f3d waiting on condition [0x00007f858c38c000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x00000005190cfe98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > > > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-146-thread-1" prio=10 tid=0x00007f85dc006800 nid=0x3f3c waiting on condition [0x00007f858c48d000]
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519018558> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > > > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-147-thread-1" prio=10 tid=0x00007f85e82b1000 nid=0x3f3b waiting on condition [0x00007f858c58e000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051903b488> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > > > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-145-thread-1" prio=10 tid=0x00007f8574027000 nid=0x3f3a waiting on condition [0x00007f858c68f000]
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051903b998> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > > > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-141-thread-1" prio=10 tid=0x00007f85e8159000 nid=0x3f39 runnable [0x00007f861cd8e000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005190db9e0> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005190db998> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000518c9c020> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-138-thread-1" prio=10 tid=0x00007f856844a800 nid=0x3f38 runnable [0x00007f861cf90000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x00000005190e1b68> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x00000005190e1b80> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x0000000518c9c608> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-143-thread-1" prio=10 tid=0x00007f85dc003000 nid=0x3f37 runnable [0x00007f861d091000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519018fa8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519018fc0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005190e2228> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "pool-137-thread-1" prio=10 tid=0x00007f8574007000 nid=0x3f36 runnable [0x00007f861ce8f000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > > > > - locked <0x0000000519003dc8> (a sun.nio.ch.Util$2)
> > > > > > - locked <0x0000000519003de0> (a java.util.Collections$UnmodifiableSet)
> > > > > > - locked <0x00000005190ea1a8> (a sun.nio.ch.EPollSelectorImpl)
> > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > > > > at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
> > > > > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "1407844613@qtp-1263282395-1 - Acceptor0 SocketConnector@0.0.0.0:54321 (http://SocketConnector@0.0.0.0:54321)" prio=10 tid=0x00007f85e404a800 nid=0x3f35 runnable [0x00007f858c790000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > > > > > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
> > > > > > - locked <0x00000005190e2720> (a java.net.SocksSocketImpl)
> > > > > > at java.net.ServerSocket.implAccept(ServerSocket.java:462)
> > > > > > at java.net.ServerSocket.accept(ServerSocket.java:430)
> > > > > > at org.mortbay.jetty.bio.SocketConnector.accept(SocketConnector.java:99)
> > > > > > at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> > > > > > at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> > > > > > 
> > > > > > "208924888@qtp-1263282395-0" prio=10 tid=0x00007f85e4021800 nid=0x3f34 in Object.wait() [0x00007f861c15b000]
> > > > > >    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> > > > > > at java.lang.Object.wait(Native Method)
> > > > > > - waiting on <0x00000005190d0528> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
> > > > > > at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> > > > > > - locked <0x00000005190d0528> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
> > > > > > 
> > > > > > "lifecycleSupervisor-1-9" prio=10 tid=0x00007f85e401a800 nid=0x3f33 waiting on condition [0x00007f861c687000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > > > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "lifecycleSupervisor-1-8" prio=10 tid=0x00007f85e4018800 nid=0x3f32 waiting on condition [0x00007f861c788000]
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > > > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "lifecycleSupervisor-1-7" prio=10 tid=0x00007f85e4016800 nid=0x3f31 waiting on condition [0x00007f861c889000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > > > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "lifecycleSupervisor-1-6" prio=10 tid=0x00007f85e4014800 nid=0x3f30 waiting on condition [0x00007f861c98a000]
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > > > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "lifecycleSupervisor-1-5" prio=10 tid=0x00007f85e4013000 nid=0x3f2f waiting on condition [0x00007f861ca8b000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > > > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "lifecycleSupervisor-1-4" prio=10 tid=0x00007f85e4028000 nid=0x3f2e waiting on condition [0x00007f861cb8c000]
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > > > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "lifecycleSupervisor-1-3" prio=10 tid=0x00007f85e4026800 nid=0x3f2d waiting on condition [0x00007f861cc8d000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > > > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Log-BackgroundWorker-ch1" prio=10 tid=0x00007f85dc001000 nid=0x3f27 waiting on condition [0x00007f861d299000]
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000519017718> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > > > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Log-BackgroundWorker-ch2" prio=10 tid=0x00007f85e80ca000 nid=0x3f26 waiting on condition [0x00007f861d39a000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x000000051903b4e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > > > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "lifecycleSupervisor-1-2" prio=10 tid=0x00007f85e4025000 nid=0x3f25 waiting on condition [0x00007f861d6ba000]
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > > > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "lifecycleSupervisor-1-1" prio=10 tid=0x00007f85e400e800 nid=0x3f24 waiting on condition [0x00007f861d7bb000] 
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > > > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "conf-file-poller-0" prio=10 tid=0x00007f85f0001000 nid=0x3f23 waiting on condition [0x00007f861d8bc000]
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000518ec72f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > > > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "DestroyJavaVM" prio=10 tid=0x00007f86a800c800 nid=0x3efa waiting on condition [0x0000000000000000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > 
> > > > > > "lifecycleSupervisor-1-0" prio=10 tid=0x00007f86a89f5800 nid=0x3f22 waiting on condition [0x00007f861d9bd000]
> > > > > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > > - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > > > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > > > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > > > > at java.lang.Thread.run(Thread.java:662)
> > > > > > 
> > > > > > "Low Memory Detector" daemon prio=10 tid=0x00007f86a831d800 nid=0x3f20 runnable [0x0000000000000000]
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > 
> > > > > > "C2 CompilerThread1" daemon prio=10 tid=0x00007f86a831b000 nid=0x3f1f waiting on condition [0x0000000000000000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > 
> > > > > > "C2 CompilerThread0" daemon prio=10 tid=0x00007f86a8319000 nid=0x3f1e waiting on condition [0x0000000000000000]
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > 
> > > > > > "Signal Dispatcher" daemon prio=10 tid=0x00007f86a8316800 nid=0x3f1d runnable [0x0000000000000000]
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > 
> > > > > > "Surrogate Locker Thread (Concurrent GC)" daemon prio=10 tid=0x00007f86a8314800 nid=0x3f1c waiting on condition [0x0000000000000000] 
> > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > 
> > > > > > "Finalizer" daemon prio=10 tid=0x00007f86a82f9800 nid=0x3f1b in Object.wait() [0x00007f861e3eb000]
> > > > > >    java.lang.Thread.State: WAITING (on object monitor)
> > > > > > at java.lang.Object.wait(Native Method)
> > > > > > - waiting on <0x0000000518c843c0> (a java.lang.ref.ReferenceQueue$Lock)
> > > > > > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> > > > > > - locked <0x0000000518c843c0> (a java.lang.ref.ReferenceQueue$Lock)
> > > > > > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> > > > > > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> > > > > > 
> > > > > > "Reference Handler" daemon prio=10 tid=0x00007f86a82f7800 nid=0x3f1a in Object.wait() [0x00007f861e4ec000]
> > > > > >    java.lang.Thread.State: WAITING (on object monitor)
> > > > > > at java.lang.Object.wait(Native Method)
> > > > > > - waiting on <0x0000000518c80860> (a java.lang.ref.Reference$Lock)
> > > > > > at java.lang.Object.wait(Object.java:485)
> > > > > > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> > > > > > - locked <0x0000000518c80860> (a java.lang.ref.Reference$Lock)
> > > > > > 
> > > > > > "VM Thread" prio=10 tid=0x00007f86a82f1000 nid=0x3f19 runnable 
> > > > > > 
> > > > > > "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f86a801b800 nid=0x3efb runnable  
> > > > > > 
> > > > > > "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f86a801d000 nid=0x3efc runnable 
> > > > > > 
> > > > > > "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f86a801f000 nid=0x3efd runnable  
> > > > > > 
> > > > > > "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f86a8021000 nid=0x3efe runnable 
> > > > > > 
> > > > > > "Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x00007f86a8022800 nid=0x3eff runnable  
> > > > > > 
> > > > > > "Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x00007f86a8024800 nid=0x3f00 runnable 
> > > > > > 
> > > > > > "Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x00007f86a8026000 nid=0x3f01 runnable  
> > > > > > 
> > > > > > "Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x00007f86a8028000 nid=0x3f02 runnable 
> > > > > > 
> > > > > > "Gang worker#8 (Parallel GC Threads)" prio=10 tid=0x00007f86a802a000 nid=0x3f03 runnable  
> > > > > > 
> > > > > > "Gang worker#9 (Parallel GC Threads)" prio=10 tid=0x00007f86a802b800 nid=0x3f04 runnable 
> > > > > > 
> > > > > > "Gang worker#10 (Parallel GC Threads)" prio=10 tid=0x00007f86a802d800 nid=0x3f05 runnable  
> > > > > > 
> > > > > > "Gang worker#11 (Parallel GC Threads)" prio=10 tid=0x00007f86a802f800 nid=0x3f06 runnable 
> > > > > > 
> > > > > > "Gang worker#12 (Parallel GC Threads)" prio=10 tid=0x00007f86a8031000 nid=0x3f07 runnable  
> > > > > > 
> > > > > > "Gang worker#13 (Parallel GC Threads)" prio=10 tid=0x00007f86a8033000 nid=0x3f08 runnable 
> > > > > > 
> > > > > > "Gang worker#14 (Parallel GC Threads)" prio=10 tid=0x00007f86a8035000 nid=0x3f09 runnable  
> > > > > > 
> > > > > > "Gang worker#15 (Parallel GC Threads)" prio=10 tid=0x00007f86a8036800 nid=0x3f0a runnable 
> > > > > > 
> > > > > > "Gang worker#16 (Parallel GC Threads)" prio=10 tid=0x00007f86a8038800 nid=0x3f0b runnable  
> > > > > > 
> > > > > > "Gang worker#17 (Parallel GC Threads)" prio=10 tid=0x00007f86a803a000 nid=0x3f0c runnable 
> > > > > > 
> > > > > > "Gang worker#18 (Parallel GC Threads)" prio=10 tid=0x00007f86a803c000 nid=0x3f0d runnable  
> > > > > > 
> > > > > > "Gang worker#19 (Parallel GC Threads)" prio=10 tid=0x00007f86a803e000 nid=0x3f0e runnable 
> > > > > > 
> > > > > > "Gang worker#20 (Parallel GC Threads)" prio=10 tid=0x00007f86a803f800 nid=0x3f0f runnable  
> > > > > > 
> > > > > > "Gang worker#21 (Parallel GC Threads)" prio=10 tid=0x00007f86a8041800 nid=0x3f10 runnable 
> > > > > > 
> > > > > > "Gang worker#22 (Parallel GC Threads)" prio=10 tid=0x00007f86a8043800 nid=0x3f11 runnable  
> > > > > > 
> > > > > > "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f86a82aa000 nid=0x3f18 runnable 
> > > > > > "Gang worker#0 (Parallel CMS Threads)" prio=10 tid=0x00007f86a829e800 nid=0x3f12 runnable 
> > > > > > 
> > > > > > "Gang worker#1 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a0000 nid=0x3f13 runnable 
> > > > > > 
> > > > > > "Gang worker#2 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a2000 nid=0x3f14 runnable  
> > > > > > 
> > > > > > "Gang worker#3 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a3800 nid=0x3f15 runnable 
> > > > > > 
> > > > > > "Gang worker#4 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a5800 nid=0x3f16 runnable  
> > > > > > 
> > > > > > "Gang worker#5 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a7800 nid=0x3f17 runnable 
> > > > > > 
> > > > > > "VM Periodic Task Thread" prio=10 tid=0x00007f86a8329000 nid=0x3f21 waiting on condition  
> > > > > > 
> > > > > > JNI global references: 1749
> > > > > > 
> > > > > >  
> > > > > > 
> > > > > > -- 
> > > > > > 
> > > > > > P | (415) 677-9222 ext. 205 (tel:%28415%29%20677-9222%20ext.%20205) F | (415) 677-0895 (tel:%28415%29%20677-0895) | pankaj@brightroll.com (mailto:pankaj@brightroll.com) 
> > > > > > Pankaj Gupta | Software Engineer
> > > > > > BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com (http://www.brightroll.com/)
> > > > > > 
> > > > > > United States | Canada | United Kingdom | Germany 
> > > > > > 
> > > > > > We're hiring (http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7)! 
> > > > > 
> > > > 
> > > > 
> > > > 
> > > > -- 
> > > > 
> > > > P | (415) 677-9222 ext. 205 (tel:%28415%29%20677-9222%20ext.%20205) F | (415) 677-0895 (tel:%28415%29%20677-0895) | pankaj@brightroll.com (mailto:pankaj@brightroll.com) 
> > > > Pankaj Gupta | Software Engineer
> > > > BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com (http://www.brightroll.com/)
> > > > 
> > > > United States | Canada | United Kingdom | Germany 
> > > > 
> > > > We're hiring (http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7)! 
> > > 
> > 
> > 
> > 
> > -- 
> > Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org 
> 
> 
> -- 
> 
> P | (415) 677-9222 ext. 205 F | (415) 677-0895 | pankaj@brightroll.com (mailto:pankaj@brightroll.com) 
> Pankaj Gupta | Software Engineer
> BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com (http://www.brightroll.com/)
> 
> United States | Canada | United Kingdom | Germany 
> 
> We're hiring (http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7)! 

Re: Lock contention in FileChannel

[Pankaj Gupta <pa...@brightroll.com>]

Looking at the code it seems like the lock and the i/o is done per event
and not for a batch. Is that correct? If that is the case then it seems
like there is a lot of overhead per event. The throughput I'm seeing is 1 -
1.5 MBps per disk which is way below the sequential read/write capacity of
the disk which is easily over 50MBps. Adding more sinks doesn't help, they
just block waiting for the queue to become free. CPU usage is 20%, there is
enough RAM for page cache so that no read is going to disk. The queue seems
to be the bottleneck. What is the throughput I should expect per disk?


On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland <br...@cloudera.com> wrote:

> The lock is per file. Adding more directories to the channel will cause
> more files to be created. Of course you'll need additional disks behind
> those directories to see any performance increase.
>
>
> On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <
> hshreedharan@cloudera.com> wrote:
>
>>  Yes, IO is done inside locks to avoid multiple takes and puts getting
>> written out at the same time. Even though Java makes sure the writes are
>> serialized, Flume still needs to keep track of some counters etc, so the
>> lock is required. Note that the lock you are talking about is  in the
>> LogFile class, which represents a single file - so even if the write is
>> inside that lock (which is also inside that class itself) that  does not
>> cause any contention - because the lock is just preventing two IO ops to
>> happen at the same time.
>>
>>
>> Thanks,
>> Hari
>>
>> On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:
>>
>> It seems like some i/o is done inside the lock, which means that time for
>> taking a lock is proportional to the time for i/o and thus it becomes a
>> problem. I apologize in advance if I am wrong but the call stack and
>> behavior I'm seeing seems to suggest that. Specifically, it seems that we
>> do a write while inside take:
>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>>    java.lang.Thread.State: RUNNABLE
>>         at sun.nio.ch.NativeThread.current(Native Method)
>>         at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>>         at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>>         - locked <0x00000005190ec998> (a java.lang.Object)
>>         at
>> org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>>         at
>> org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>>         - locked <0x0000000519111590> (a
>> org.apache.flume.channel.file.LogFileV3$Writer)
>>         at org.apache.flume.channel.file.Log.take(Log.java:550)
>>         at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>>         at
>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>         at
>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>         at
>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>         at
>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>         at
>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>         at java.lang.Thread.run(Thread.java:662)
>>
>>
>>
>> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <
>> hshreedharan@cloudera.com> wrote:
>>
>> Since the channel is designed to make sure that events are not duplicated
>> to multiple sinks, and to protect against corruption due to concurrency
>> issues, we do not need the locking in the channel's flume event queue. It
>> is unlikely that locking is what is causing performance issues because the
>> channel is heavily I/O bound. If you take a series of thread dumps, you
>> will probably see that those threads are moving forward and the ones
>> reading/writing from/to disk are the ones which are slower. These locks are
>> unlikely to hit performance much.
>>
>> Thanks,
>> Hari
>>
>> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
>>
>> Hi,
>>
>> Spent some more time debugging issues with FileChannel. The problem seems
>> to lock contention reading from FlumeEventQueue:
>>
>> I see a lot of threads like this:
>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
>> [0x00007f821afee000]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>         at
>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>>         - waiting to lock <0x0000000518ee4c90> (a
>> org.apache.flume.channel.file.FlumeEventQueue)
>>         at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>>         at
>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>         at
>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>         at
>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>         at
>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>         at
>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>         at java.lang.Thread.run(Thread.java:662)
>>
>>
>> I have two file channels and 8 Avro Sinks per file channel. I added more
>> sinks because they weren't draining fast enough. It seems like they send
>> the batch then wait for an ack before sending again, thus sends are not
>> pipelined and having more sinks seemed like a good way of getting some
>> parallelism.
>>
>> Here's the full stack trace:
>> 2013-08-13 15:30:32
>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02 mixed mode):
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
>>    java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000056d932120> (a
>> java.util.concurrent.CountDownLatch$Sync)
>>  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>>  at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>>  at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
>> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
>>  at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
>> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
>>  at
>> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
>> at $Proxy7.appendBatch(Unknown Source)
>>  at
>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
>> at
>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
>>  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000 nid=0x41a1
>> waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>  at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005194c3340> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005191a16d8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519174e08> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519174e98> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005195f9670> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519174ce8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005191a1918> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005191e7f60> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153 runnable
>> [0x00007f8210e4d000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
>> - locked <0x000000054a2917a8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519174f58> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146 runnable
>> [0x00007f8210f4e000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
>> - locked <0x0000000546e5dd28> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118 runnable
>> [0x00007f821104f000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000544f763c0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111 runnable
>> [0x00007f8211150000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
>> - locked <0x000000054473d5a8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110 runnable
>> [0x00007f8211251000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
>> - locked <0x00000005437f4f70> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f runnable
>> [0x00007f8211352000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
>> - locked <0x00000005422a9018> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e runnable
>> [0x00007f8211453000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
>> - locked <0x0000000541063d50> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d runnable
>> [0x00007f8211554000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
>> - locked <0x0000000540dc1110> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a runnable
>> [0x00007f8211655000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
>> - locked <0x000000053fbbc3b0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109 runnable
>> [0x00007f8211756000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
>> - locked <0x000000053f812860> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd runnable
>> [0x00007f8211857000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
>> - locked <0x000000053e6e5570> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519175138> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519174c28> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005191751c8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0 runnable
>> [0x00007f8211c5b000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000536579db0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005191750a8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1 runnable
>> [0x00007f8211e5d000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
>> - locked <0x00000005313943a8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051936f460> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051936dfa8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de runnable
>> [0x00007f8212160000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
>> - locked <0x000000052dad4468> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd runnable
>> [0x00007f8212261000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
>> - locked <0x000000052c73c5f0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc runnable
>> [0x00007f8212362000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
>> - locked <0x000000052b8da998> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db waiting
>> for monitor entry [0x00007f8212462000]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>> - waiting to lock <0x0000000518ee4c90> (a
>> org.apache.flume.channel.file.FlumeEventQueue)
>>  at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>> at
>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>  at
>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>> at
>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>  at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>>  at
>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>  at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>  at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>  at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>  at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>  at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>  at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051936df18> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>> tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000518fe7ae8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8 runnable
>> [0x00007f8212766000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
>> - locked <0x00000005263d6ba8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7 runnable
>> [0x00007f8212867000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000525d1d0e0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6 runnable
>> [0x00007f8212968000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000525a689d8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5 runnable
>> [0x00007f8212a69000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000524e1e7d8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4 runnable
>> [0x00007f8212b6a000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000524d437a8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-36" prio=10 tid=0x00007f85843ad800 nid=0x40d3 runnable
>> [0x00007f8212c6b000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000524cf4140> (a sun.nio.ch.Util$2)
>> - locked <0x0000000524cf4128> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000524ab32e8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-35" prio=10 tid=0x00007f85d83a9000 nid=0x40d2 runnable
>> [0x00007f8212d6c000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005244877c8> (a sun.nio.ch.Util$2)
>> - locked <0x00000005244877b0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000523f5b660> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-34" prio=10 tid=0x00007f85d838e000 nid=0x40d1 runnable
>> [0x00007f8212e6d000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000052419e490> (a sun.nio.ch.Util$2)
>> - locked <0x000000052419e478> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000523f5b6f0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-34" prio=10 tid=0x00007f8588390800 nid=0x40d0 waiting
>> for monitor entry [0x00007f8212f6d000]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
>> - waiting to lock <0x0000000519111590> (a
>> org.apache.flume.channel.file.LogFileV3$Writer)
>>  at org.apache.flume.channel.file.Log.put(Log.java:507)
>> at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
>>  at
>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>> at
>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>  at
>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>  at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>  at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>> at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>  at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>> at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>  at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>  at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>  at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>  at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-35" prio=10 tid=0x00007f85843ab800 nid=0x40cf runnable
>> [0x00007f821306f000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000523d40938> (a sun.nio.ch.Util$2)
>> - locked <0x0000000523d40aa0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000523ecec18> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-37" prio=10 tid=0x00007f85803e3800 nid=0x40ce waiting
>> for monitor entry [0x00007f821316f000]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>> - waiting to lock <0x0000000518ee4c90> (a
>> org.apache.flume.channel.file.FlumeEventQueue)
>>  at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>> at
>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>  at
>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>> at
>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>  at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>>  at
>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>  at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>  at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>  at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>  at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>  at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-34" prio=10 tid=0x00007f8584391000 nid=0x40cd runnable
>> [0x00007f8213271000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> - locked <0x0000000523935540> (a sun.nio.ch.Util$2)
>>  - locked <0x0000000523935528> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x0000000523788ed8> (a sun.nio.ch.EPollSelectorImpl)
>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-33" prio=10 tid=0x00007f85d8373800 nid=0x40cc runnable
>> [0x00007f8213372000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> - locked <0x00000005239ec658> (a sun.nio.ch.Util$2)
>>  - locked <0x00000005239ec640> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x0000000523845410> (a sun.nio.ch.EPollSelectorImpl)
>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-33" prio=10 tid=0x00007f8588375800 nid=0x40cb waiting
>> for monitor entry [0x00007f8213472000]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
>>  - waiting to lock <0x0000000519111590> (a
>> org.apache.flume.channel.file.LogFileV3$Writer)
>> at org.apache.flume.channel.file.Log.put(Log.java:507)
>>  at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
>> at
>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>  at
>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>> at
>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>  at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>>  at
>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>  at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>  at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>  at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>  at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>  at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-33" prio=10 tid=0x00007f8584376000 nid=0x40ca runnable
>> [0x00007f8213574000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> - locked <0x00000005237e3028> (a sun.nio.ch.Util$2)
>>  - locked <0x00000005237e3010> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x0000000523788e90> (a sun.nio.ch.EPollSelectorImpl)
>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-32" prio=10 tid=0x00007f85d8358800 nid=0x40c9 waiting
>> for monitor entry [0x00007f8213674000]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>>  - waiting to lock <0x0000000518ee4c90> (a
>> org.apache.flume.channel.file.FlumeEventQueue)
>> at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>>  at
>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>> at
>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>  at
>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>  at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>  at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>> at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>  at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>> at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>  at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>  at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>  at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>  at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-32" prio=10 tid=0x00007f858835b000 nid=0x40c8 runnable
>> [0x00007f8213776000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000523343688> (a sun.nio.ch.Util$2)
>> - locked <0x0000000523343670> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000052305e5c8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-36" prio=10 tid=0x00007f85803c9000 nid=0x40c7 runnable
>> [0x00007f8213877000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005230fa580> (a sun.nio.ch.Util$2)
>> - locked <0x00000005230fa568> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000522f5ca50> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-32" prio=10 tid=0x00007f858435b000 nid=0x40c6 runnable
>> [0x00007f8213978000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005231367a8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000523136790> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000052305e658> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-31" prio=10 tid=0x00007f85d833e000 nid=0x40c5 runnable
>> [0x00007f8213a79000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000522db30a0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000522db3088> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000522bda430> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-31" prio=10 tid=0x00007f8588340000 nid=0x40c4 runnable
>> [0x00007f8213b7a000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000522c16518> (a sun.nio.ch.Util$2)
>> - locked <0x0000000522c16500> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000522bd97e8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-30" prio=10 tid=0x00007f85d8323800 nid=0x40c3 runnable
>> [0x00007f8213c7b000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000522bd1a20> (a sun.nio.ch.Util$2)
>> - locked <0x0000000522bd1a08> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000522bda3a0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-31" prio=10 tid=0x00007f8584340800 nid=0x40c2 runnable
>> [0x00007f8213d7c000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005228d1608> (a sun.nio.ch.Util$2)
>> - locked <0x00000005228d15f0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005228a6aa0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-30" prio=10 tid=0x00007f8588325000 nid=0x40c1 runnable
>> [0x00007f8213e7d000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000052277b088> (a sun.nio.ch.Util$2)
>> - locked <0x000000052277b070> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005225ed3b8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-35" prio=10 tid=0x00007f85803ae000 nid=0x40c0 runnable
>> [0x00007f8213f7e000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000522737380> (a sun.nio.ch.Util$2)
>> - locked <0x0000000522737368> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005225ed328> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-29" prio=10 tid=0x00007f85d8308800 nid=0x40bf runnable
>> [0x00007f821407f000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005226034a8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000522603490> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005225ed298> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-28" prio=10 tid=0x00007f85d82ed800 nid=0x40be runnable
>> [0x00007f8214180000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000522359408> (a sun.nio.ch.Util$2)
>> - locked <0x00000005223593f0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005221e7278> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-27" prio=10 tid=0x00007f85d82d2800 nid=0x40bd runnable
>> [0x00007f8214281000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005222c0478> (a sun.nio.ch.Util$2)
>> - locked <0x00000005222c0460> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005221e6e40> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-29" prio=10 tid=0x00007f858830a800 nid=0x40bc runnable
>> [0x00007f8214382000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000521ddf488> (a sun.nio.ch.Util$2)
>> - locked <0x0000000521ddf470> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000521c88d38> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-30" prio=10 tid=0x00007f8584326000 nid=0x40bb waiting
>> for monitor entry [0x00007f8214482000]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
>> - waiting to lock <0x0000000519111590> (a
>> org.apache.flume.channel.file.LogFileV3$Writer)
>>  at org.apache.flume.channel.file.Log.put(Log.java:507)
>> at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
>>  at
>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>> at
>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>  at
>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>  at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>  at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>> at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>  at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>> at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>  at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>  at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>  at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>  at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-26" prio=10 tid=0x00007f85d82b8000 nid=0x40ba runnable
>> [0x00007f8214584000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000521cbcb90> (a sun.nio.ch.Util$2)
>> - locked <0x0000000521cbcb78> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000521279250> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-28" prio=10 tid=0x00007f85882ef800 nid=0x40b9 runnable
>> [0x00007f8214685000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000521a1c2d0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000521a1c2b8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005218f1508> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-34" prio=10 tid=0x00007f8580393800 nid=0x40b8 runnable
>> [0x00007f8214786000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000052181ce58> (a sun.nio.ch.Util$2)
>> - locked <0x000000052181ce40> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005212791c0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-29" prio=10 tid=0x00007f858430b800 nid=0x40b7 runnable
>> [0x00007f8214887000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005219175f0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000521746090> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000521278e60> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-27" prio=10 tid=0x00007f85882d4800 nid=0x40b6 runnable
>> [0x00007f8214988000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000052163e9c8> (a sun.nio.ch.Util$2)
>> - locked <0x000000052163e9b0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000521279130> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-33" prio=10 tid=0x00007f8580378800 nid=0x40b5 runnable
>> [0x00007f8214a89000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000052152cb38> (a sun.nio.ch.Util$2)
>> - locked <0x000000052152cb20> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000521278d40> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-28" prio=10 tid=0x00007f85842f0800 nid=0x40b4 runnable
>> [0x00007f8214b8a000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005213ba3e0> (a sun.nio.ch.Util$2)
>> - locked <0x00000005213ba3c8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000521278020> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-27" prio=10 tid=0x00007f85842d5800 nid=0x40b3 runnable
>> [0x00007f8214c8b000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005213ed660> (a sun.nio.ch.Util$2)
>> - locked <0x00000005213ed648> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000521278c20> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-26" prio=10 tid=0x00007f85882b9800 nid=0x40b2 runnable
>> [0x00007f8214d8c000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000052133b7c0> (a sun.nio.ch.Util$2)
>> - locked <0x000000052133b7a8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005212790a0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-32" prio=10 tid=0x00007f858035d800 nid=0x40af runnable
>> [0x00007f8214e8d000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000520e53be0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000520e53bc8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000520f3bb50> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-25" prio=10 tid=0x00007f85d829d000 nid=0x40ae runnable
>> [0x00007f8214f8e000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000520e09a30> (a sun.nio.ch.Util$2)
>> - locked <0x0000000520e09a18> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000520f3b6d0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-31" prio=10 tid=0x00007f8580342800 nid=0x40ad runnable
>> [0x00007f821508f000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000520ef70e0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000520ef70c8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000520f3b208> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-26" prio=10 tid=0x00007f85842ba800 nid=0x40ac runnable
>> [0x00007f8215190000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000520c9dfc0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000520c9dfa8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000520bb3cc0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-25" prio=10 tid=0x00007f85842a0000 nid=0x40ab runnable
>> [0x00007f8215291000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000520b941a8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000520b94190> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000520bb30c0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-24" prio=10 tid=0x00007f85d8282800 nid=0x40aa runnable
>> [0x00007f8215392000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000520952288> (a sun.nio.ch.Util$2)
>> - locked <0x0000000520952270> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000520356888> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-30" prio=10 tid=0x00007f8580328000 nid=0x40a9 runnable
>> [0x00007f8215493000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000520a2fa20> (a sun.nio.ch.Util$2)
>> - locked <0x0000000520a2fa08> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000520983da0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-23" prio=10 tid=0x00007f85d8267800 nid=0x40a8 runnable
>> [0x00007f8215594000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000052095c470> (a sun.nio.ch.Util$2)
>> - locked <0x000000052095c458> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000520356840> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-29" prio=10 tid=0x00007f858030d000 nid=0x40a7 runnable
>> [0x00007f8215695000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000052065b178> (a sun.nio.ch.Util$2)
>> - locked <0x000000052065b160> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000520357320> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-28" prio=10 tid=0x00007f85802f2800 nid=0x40a6 runnable
>> [0x00007f8215796000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000052043cad0> (a sun.nio.ch.Util$2)
>> - locked <0x000000052043cab8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005203567b0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-25" prio=10 tid=0x00007f858829f000 nid=0x40a5 runnable
>> [0x00007f8215897000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000052041a8a8> (a sun.nio.ch.Util$2)
>> - locked <0x000000052041a890> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000520356690> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-22" prio=10 tid=0x00007f85d824c800 nid=0x40a4 runnable
>> [0x00007f8215998000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000052036b380> (a sun.nio.ch.Util$2)
>> - locked <0x000000052036b368> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000520357290> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-24" prio=10 tid=0x00007f8584285800 nid=0x40a3 runnable
>> [0x00007f8215a99000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000052007ac40> (a sun.nio.ch.Util$2)
>> - locked <0x000000052007ac28> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051fe41948> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-27" prio=10 tid=0x00007f85802d8000 nid=0x40a2 runnable
>> [0x00007f8215b9a000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005201a2ae8> (a sun.nio.ch.Util$2)
>> - locked <0x00000005201a2ad0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051ffda458> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-21" prio=10 tid=0x00007f85d8232000 nid=0x40a1 waiting
>> for monitor entry [0x00007f8215c9a000]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>> - waiting to lock <0x0000000518ee4c90> (a
>> org.apache.flume.channel.file.FlumeEventQueue)
>>  at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>> at
>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>  at
>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>> at
>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>  at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>>  at
>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>  at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>  at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>  at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>  at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>  at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-26" prio=10 tid=0x00007f85802bd800 nid=0x40a0 runnable
>> [0x00007f8215d9c000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> - locked <0x000000051ff20e78> (a sun.nio.ch.Util$2)
>>  - locked <0x000000051ff20e60> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000051fe41900> (a sun.nio.ch.EPollSelectorImpl)
>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-24" prio=10 tid=0x00007f8588284000 nid=0x409f runnable
>> [0x00007f8215e9d000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> - locked <0x000000051fbc1ca0> (a sun.nio.ch.Util$2)
>>  - locked <0x000000051fbc1c88> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000051f9a7978> (a sun.nio.ch.EPollSelectorImpl)
>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-25" prio=10 tid=0x00007f85802a2800 nid=0x409e runnable
>> [0x00007f8215f9e000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> - locked <0x000000051fa3b5b8> (a sun.nio.ch.Util$2)
>>  - locked <0x000000051fa3b5a0> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000051f3155b0> (a sun.nio.ch.EPollSelectorImpl)
>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-20" prio=10 tid=0x00007f85d8217800 nid=0x409d runnable
>> [0x00007f821609f000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> - locked <0x000000051f83c838> (a sun.nio.ch.Util$2)
>>  - locked <0x000000051f83c820> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000051f3150e8> (a sun.nio.ch.EPollSelectorImpl)
>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-23" prio=10 tid=0x00007f8588269800 nid=0x409c runnable
>> [0x00007f82161a0000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> - locked <0x000000051f78b9b0> (a sun.nio.ch.Util$2)
>>  - locked <0x000000051f78b998> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000051f3156d0> (a sun.nio.ch.EPollSelectorImpl)
>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-22" prio=10 tid=0x00007f858824e800 nid=0x409b runnable
>> [0x00007f82162a1000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> - locked <0x000000051f7c0cc8> (a sun.nio.ch.Util$2)
>>  - locked <0x000000051f7c0cb0> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000051f3151c0> (a sun.nio.ch.EPollSelectorImpl)
>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-23" prio=10 tid=0x00007f858426a800 nid=0x409a runnable
>> [0x00007f82163a2000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> - locked <0x000000051f6065a8> (a sun.nio.ch.Util$2)
>>  - locked <0x000000051f606590> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000051f3d9668> (a sun.nio.ch.EPollSelectorImpl)
>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-22" prio=10 tid=0x00007f858424f800 nid=0x4099 runnable
>> [0x00007f82164a3000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> - locked <0x000000051f56d710> (a sun.nio.ch.Util$2)
>>  - locked <0x000000051f56d6f8> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000051f314f80> (a sun.nio.ch.EPollSelectorImpl)
>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-21" prio=10 tid=0x00007f8588233800 nid=0x4098 runnable
>> [0x00007f82165a4000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> - locked <0x000000051f3bcd68> (a sun.nio.ch.Util$2)
>>  - locked <0x000000051f3bcd50> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000051f315400> (a sun.nio.ch.EPollSelectorImpl)
>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-19" prio=10 tid=0x00007f85d81fc800 nid=0x4097 runnable
>> [0x00007f82166a5000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> - locked <0x000000051f53b198> (a sun.nio.ch.Util$2)
>>  - locked <0x000000051f53b180> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000051f315760> (a sun.nio.ch.EPollSelectorImpl)
>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-20" prio=10 tid=0x00007f8588218800 nid=0x4096 waiting
>> for monitor entry [0x00007f82167a5000]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
>>  - waiting to lock <0x0000000519111590> (a
>> org.apache.flume.channel.file.LogFileV3$Writer)
>> at org.apache.flume.channel.file.Log.put(Log.java:507)
>>  at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
>> at
>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>  at
>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>> at
>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>  at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>>  at
>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>  at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>  at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>  at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>  at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>> at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>  at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-18" prio=10 tid=0x00007f85d81e1800 nid=0x4095 runnable
>> [0x00007f82168a7000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> - locked <0x000000051ee92ba0> (a sun.nio.ch.Util$2)
>>  - locked <0x000000051ee92b88> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000051ed19740> (a sun.nio.ch.EPollSelectorImpl)
>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-19" prio=10 tid=0x00007f85881fd800 nid=0x4094 runnable
>> [0x00007f82169a8000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> - locked <0x000000051ee5da40> (a sun.nio.ch.Util$2)
>>  - locked <0x000000051ee5da28> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000051e77a940> (a sun.nio.ch.EPollSelectorImpl)
>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-17" prio=10 tid=0x00007f85d81c6800 nid=0x4093 runnable
>> [0x00007f8216aa9000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> - locked <0x000000051ecb1730> (a sun.nio.ch.Util$2)
>>  - locked <0x000000051ecb1718> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000051e77a8b0> (a sun.nio.ch.EPollSelectorImpl)
>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-21" prio=10 tid=0x00007f8584234800 nid=0x4092 runnable
>> [0x00007f8216baa000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> - locked <0x000000051e899660> (a sun.nio.ch.Util$2)
>>  - locked <0x000000051e899648> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x000000051e77a310> (a sun.nio.ch.EPollSelectorImpl)
>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-24" prio=10 tid=0x00007f8580287000 nid=0x4091 runnable
>> [0x00007f8216caa000]
>>    java.lang.Thread.State: RUNNABLE
>> at
>> org.apache.flume.channel.file.FlumeEventQueue.addTail(FlumeEventQueue.java:165)
>>  - locked <0x0000000518ee4c90> (a
>> org.apache.flume.channel.file.FlumeEventQueue)
>> at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:529)
>>  - locked <0x0000000518ee4c90> (a
>> org.apache.flume.channel.file.FlumeEventQueue)
>> at
>> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
>>  at
>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
>> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>  at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>  at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>> at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>  at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>> at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>  at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>  at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>  at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>  at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-23" prio=10 tid=0x00007f858026c000 nid=0x4090 runnable
>> [0x00007f8216dac000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051ea8b6f0> (a sun.nio.ch.Util$2)
>> - locked <0x000000051ea8b6d8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051e77a820> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-16" prio=10 tid=0x00007f85d81ac000 nid=0x408f runnable
>> [0x00007f8216ead000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051e7c8808> (a sun.nio.ch.Util$2)
>> - locked <0x000000051e7c87f0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051e77a280> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-20" prio=10 tid=0x00007f8584219800 nid=0x408e runnable
>> [0x00007f8216fae000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051e8fffa8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051e8fff90> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051e77a700> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-22" prio=10 tid=0x00007f8580251000 nid=0x408d runnable
>> [0x00007f82170af000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051e4dc398> (a sun.nio.ch.Util$2)
>> - locked <0x000000051e4dc380> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051dcbeef8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-19" prio=10 tid=0x00007f85841fe800 nid=0x408c runnable
>> [0x00007f82171b0000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051e221520> (a sun.nio.ch.Util$2)
>> - locked <0x000000051e221508> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051e407860> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-18" prio=10 tid=0x00007f85841e3800 nid=0x408b runnable
>> [0x00007f82172b1000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051e465298> (a sun.nio.ch.Util$2)
>> - locked <0x000000051e465280> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051dcbef88> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-18" prio=10 tid=0x00007f85881e3000 nid=0x408a runnable
>> [0x00007f82173b2000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051dea12e0> (a sun.nio.ch.Util$2)
>> - locked <0x000000051dea12c8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051cfd4d78> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-15" prio=10 tid=0x00007f85d8191000 nid=0x4089 runnable
>> [0x00007f82174b3000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051de322b8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051de322a0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051dcbec28> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-14" prio=10 tid=0x00007f85d8176800 nid=0x4088 runnable
>> [0x00007f82175b4000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051dea29a8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051dea2990> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051dcbf018> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-21" prio=10 tid=0x00007f8580236800 nid=0x4087 runnable
>> [0x00007f82176b5000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051dca95d0> (a sun.nio.ch.Util$2)
>> - locked <0x000000051dca95b8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051cfd4e08> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-17" prio=10 tid=0x00007f85841c8800 nid=0x4086 runnable
>> [0x00007f82177b6000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051da8d058> (a sun.nio.ch.Util$2)
>> - locked <0x000000051da8d040> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051cd693f8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-20" prio=10 tid=0x00007f858021b800 nid=0x4085 runnable
>> [0x00007f82178b7000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051d95a5b0> (a sun.nio.ch.Util$2)
>> - locked <0x000000051d95a598> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051cfd4e98> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-16" prio=10 tid=0x00007f85841ad800 nid=0x4083 runnable
>> [0x00007f82179b8000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051d92a278> (a sun.nio.ch.Util$2)
>> - locked <0x000000051d92a260> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051cfd4fb8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-13" prio=10 tid=0x00007f85d815b800 nid=0x4082 runnable
>> [0x00007f8217ab9000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051d734788> (a sun.nio.ch.Util$2)
>> - locked <0x000000051d734770> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051cd69488> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-19" prio=10 tid=0x00007f8580200800 nid=0x4081 runnable
>> [0x00007f8217bba000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051d4268e8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051d4268d0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051d139810> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-15" prio=10 tid=0x00007f8584192800 nid=0x4080 runnable
>> [0x00007f8217cbb000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051d4cda88> (a sun.nio.ch.Util$2)
>> - locked <0x000000051d4cda70> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051babcf98> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-17" prio=10 tid=0x00007f85881c8000 nid=0x407f runnable
>> [0x00007f8217dbc000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051d32ffe8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051d32ffd0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051cd691b8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-14" prio=10 tid=0x00007f8584177800 nid=0x407e runnable
>> [0x00007f8217ebd000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051d506c60> (a sun.nio.ch.Util$2)
>> - locked <0x000000051d506c48> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051cfd5048> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-16" prio=10 tid=0x00007f85881ad000 nid=0x407d runnable
>> [0x00007f8217fbe000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051d1223e0> (a sun.nio.ch.Util$2)
>> - locked <0x000000051d1223c8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051c6c7b40> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-15" prio=10 tid=0x00007f8588192800 nid=0x407c runnable
>> [0x00007f82180bf000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051d146f50> (a sun.nio.ch.Util$2)
>> - locked <0x000000051d146f38> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051cd69248> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-13" prio=10 tid=0x00007f858415c800 nid=0x407b runnable
>> [0x00007f82181c0000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051d11d1b8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051d11d1a0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051c6c7c18> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-12" prio=10 tid=0x00007f85d8141000 nid=0x407a runnable
>> [0x00007f82182c1000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051cd01a78> (a sun.nio.ch.Util$2)
>> - locked <0x000000051cd01a60> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051cd695a8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-11" prio=10 tid=0x00007f85d8126800 nid=0x4079 runnable
>> [0x00007f82183c2000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051cfa6f88> (a sun.nio.ch.Util$2)
>> - locked <0x000000051cfa6f70> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051ba67370> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-14" prio=10 tid=0x00007f8588177800 nid=0x4078 runnable
>> [0x00007f82184c3000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051ce5cdd0> (a sun.nio.ch.Util$2)
>> - locked <0x000000051ce5cdb8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051c6c7bd0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-12" prio=10 tid=0x00007f8584142000 nid=0x4077 runnable
>> [0x00007f82185c4000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051cc27cb8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051cc27ca0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051babd0b8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-18" prio=10 tid=0x00007f85801e6000 nid=0x4076 runnable
>> [0x00007f82186c5000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051cb895c8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051cc77fd0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051c6c7cf0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-11" prio=10 tid=0x00007f8584127800 nid=0x4075 runnable
>> [0x00007f82187c6000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051ca540d0> (a sun.nio.ch.Util$2)
>> - locked <0x000000051caad368> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051c6c7e10> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-13" prio=10 tid=0x00007f858815c800 nid=0x4074 runnable
>> [0x00007f82188c7000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051c953ab8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051ca799a8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051babd028> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-17" prio=10 tid=0x00007f85801cb800 nid=0x4073 runnable
>> [0x00007f82189c8000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051c77c3d8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051c8137c8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051c3856f0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-10" prio=10 tid=0x00007f858410c800 nid=0x4072 runnable
>> [0x00007f8218ac9000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051c7dd2b0> (a sun.nio.ch.Util$2)
>> - locked <0x000000051c613988> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051babcde8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-16" prio=10 tid=0x00007f85801b1000 nid=0x4071 runnable
>> [0x00007f8218bca000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051c776580> (a sun.nio.ch.Util$2)
>> - locked <0x000000051c6633c8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051c6c7ea0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-9" prio=10 tid=0x00007f85840f1800 nid=0x4070 runnable
>> [0x00007f8218ccb000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051c36b550> (a sun.nio.ch.Util$2)
>> - locked <0x000000051c36b568> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051ba675f8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-8" prio=10 tid=0x00007f85840d6800 nid=0x406f runnable
>> [0x00007f8218dcc000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051c546b00> (a sun.nio.ch.Util$2)
>> - locked <0x000000051c40a178> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051ba67760> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-7" prio=10 tid=0x00007f85840bb800 nid=0x406e runnable
>> [0x00007f8218ecd000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051c2ecf90> (a sun.nio.ch.Util$2)
>> - locked <0x000000051c29a100> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051ba675b0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-15" prio=10 tid=0x00007f8580196000 nid=0x406d runnable
>> [0x00007f8218fce000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051c2ee670> (a sun.nio.ch.Util$2)
>> - locked <0x000000051c2ee688> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051ba67718> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-12" prio=10 tid=0x00007f8588141800 nid=0x406c runnable
>> [0x00007f82190cf000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051c0bd978> (a sun.nio.ch.Util$2)
>> - locked <0x000000051c16cc00> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051babcf08> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-6" prio=10 tid=0x00007f85840a0800 nid=0x406b runnable
>> [0x00007f82191d0000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051c068508> (a sun.nio.ch.Util$2)
>> - locked <0x000000051bf825f0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051ba676d0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-11" prio=10 tid=0x00007f8588127000 nid=0x406a runnable
>> [0x00007f82192d1000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051becaa68> (a sun.nio.ch.Util$2)
>> - locked <0x000000051c03ead0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051babce78> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-10" prio=10 tid=0x00007f858810c800 nid=0x4069 runnable
>> [0x00007f82193d2000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051be31b98> (a sun.nio.ch.Util$2)
>> - locked <0x000000051bd56790> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051ba67490> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-14" prio=10 tid=0x00007f858017b000 nid=0x4068 runnable
>> [0x00007f82194d3000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051bccb8c0> (a sun.nio.ch.Util$2)
>> - locked <0x000000051bb14420> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051babcd58> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-5" prio=10 tid=0x00007f8584085800 nid=0x4067 runnable
>> [0x00007f82195d4000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051bd004a8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051bd49480> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051ba67400> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-4" prio=10 tid=0x00007f858406b000 nid=0x4066 runnable
>> [0x00007f82196d5000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051ba1e928> (a sun.nio.ch.Util$2)
>> - locked <0x000000051bccdd00> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051ba66ef0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-9" prio=10 tid=0x00007f85880f1800 nid=0x4065 waiting for
>> monitor entry [0x00007f82197d5000]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
>> - waiting to lock <0x0000000519111590> (a
>> org.apache.flume.channel.file.LogFileV3$Writer)
>>  at org.apache.flume.channel.file.Log.put(Log.java:507)
>> at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
>>  at
>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>> at
>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>  at
>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>  at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>  at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>> at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>  at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>> at
>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>  at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>  at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>> at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>  at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>> at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>  at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>  at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-8" prio=10 tid=0x00007f85880d6800 nid=0x4064 runnable
>> [0x00007f82198d7000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051b5bcca8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051b7ef8f0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051b5c7cb8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-7" prio=10 tid=0x00007f85880bb800 nid=0x4063 runnable
>> [0x00007f82199d8000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051b788ce8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051b788d00> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051b285260> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-13" prio=10 tid=0x00007f8580160800 nid=0x4062 runnable
>> [0x00007f8219ad9000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051b71a0f8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051b750e98> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051aed5a70> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-10" prio=10 tid=0x00007f85d810b800 nid=0x4061 runnable
>> [0x00007f8219bda000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051b517878> (a sun.nio.ch.Util$2)
>> - locked <0x000000051b5bab30> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051aed5b90> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-9" prio=10 tid=0x00007f85d80f0800 nid=0x4060 runnable
>> [0x00007f8219cdb000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051b5656e0> (a sun.nio.ch.Util$2)
>> - locked <0x000000051b5657a0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051b285338> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-8" prio=10 tid=0x00007f85d80d5800 nid=0x405f runnable
>> [0x00007f8219ddc000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051b3df220> (a sun.nio.ch.Util$2)
>> - locked <0x000000051b475118> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051aed5c20> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-6" prio=10 tid=0x00007f85880a0800 nid=0x405e runnable
>> [0x00007f8219edd000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051b11c010> (a sun.nio.ch.Util$2)
>> - locked <0x000000051b1eab78> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051af91478> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-7" prio=10 tid=0x00007f85d80bb000 nid=0x405d runnable
>> [0x00007f8219fde000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051b1d36e0> (a sun.nio.ch.Util$2)
>> - locked <0x000000051b0328c0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051aed5cb0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-12" prio=10 tid=0x00007f8580145800 nid=0x405c runnable
>> [0x00007f821a0df000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051aeb3070> (a sun.nio.ch.Util$2)
>> - locked <0x000000051aeb3088> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051a98a820> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-6" prio=10 tid=0x00007f85d80a0000 nid=0x405b runnable
>> [0x00007f821a1e0000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051aefa920> (a sun.nio.ch.Util$2)
>> - locked <0x000000051af3ee40> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051ac28810> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-5" prio=10 tid=0x00007f8588086000 nid=0x405a runnable
>> [0x00007f821a2e1000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051aefe1b0> (a sun.nio.ch.Util$2)
>> - locked <0x000000051ad2b858> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051aed5e18> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-11" prio=10 tid=0x00007f858012a800 nid=0x4059 runnable
>> [0x00007f821a3e2000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051ac42588> (a sun.nio.ch.Util$2)
>> - locked <0x000000051ad05d70> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051a98a7d8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-3" prio=10 tid=0x00007f8584050000 nid=0x4058 runnable
>> [0x00007f821a4e3000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051ad06100> (a sun.nio.ch.Util$2)
>> - locked <0x000000051ad47428> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051ac289c0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-10" prio=10 tid=0x00007f858010f800 nid=0x4057 runnable
>> [0x00007f821a5e4000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051aab5d80> (a sun.nio.ch.Util$2)
>> - locked <0x000000051ab8ce38> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051a98a748> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-5" prio=10 tid=0x00007f85d8085800 nid=0x4056 runnable
>> [0x00007f821a6e5000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051a962980> (a sun.nio.ch.Util$2)
>> - locked <0x000000051a775fb0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051a842098> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-2" prio=10 tid=0x00007f8584035000 nid=0x4055 runnable
>> [0x00007f821a7e6000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051a817210> (a sun.nio.ch.Util$2)
>> - locked <0x000000051a7d2b48> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051a841ca8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-4" prio=10 tid=0x00007f85d806b000 nid=0x4054 runnable
>> [0x00007f821a8e7000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051a903340> (a sun.nio.ch.Util$2)
>> - locked <0x000000051a903358> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051a42db50> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-9" prio=10 tid=0x00007f85800f4800 nid=0x4053 runnable
>> [0x00007f821a9e8000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051a4f17d8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051a37ae08> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051a22e340> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-4" prio=10 tid=0x00007f858806b000 nid=0x4052 runnable
>> [0x00007f821aae9000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051a4d81e8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051a616bb0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051a1dfb40> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-3" prio=10 tid=0x00007f85d8050000 nid=0x4051 runnable
>> [0x00007f821abea000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051a4fb240> (a sun.nio.ch.Util$2)
>> - locked <0x000000051a3913b8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051a213878> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-2" prio=10 tid=0x00007f85d8035800 nid=0x4050 runnable
>> [0x00007f821aceb000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051a506248> (a sun.nio.ch.Util$2)
>> - locked <0x000000051a506308> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051a42dd48> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-8" prio=10 tid=0x00007f85800d9800 nid=0x404f runnable
>> [0x00007f821adec000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051a2d5170> (a sun.nio.ch.Util$2)
>> - locked <0x000000051a2d5188> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051a22e2b0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-7" prio=10 tid=0x00007f85800be800 nid=0x404e runnable
>> [0x00007f821aeed000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051a2d3cb8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051a2b8310> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051a213758> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
>> [0x00007f821afee000]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>> - waiting to lock <0x0000000518ee4c90> (a
>> org.apache.flume.channel.file.FlumeEventQueue)
>>  at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>> at
>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>  at
>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>> at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>  at
>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f836c01a000 nid=0x404c runnable [0x00007f821b0ef000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051a045bb0> (a sun.nio.ch.Util$2)
>> - locked <0x000000051a045bc8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051a000c98> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b376800
>> nid=0x404b waiting on condition [0x00007f821b1f0000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051a045c10> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.NativeThread.current(Native Method)
>> at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>>  at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>> - locked <0x00000005190ec998> (a java.lang.Object)
>>  at org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>> at org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>>  - locked <0x0000000519111590> (a
>> org.apache.flume.channel.file.LogFileV3$Writer)
>> at org.apache.flume.channel.file.Log.take(Log.java:550)
>>  at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>> at
>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>  at
>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>> at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>  at
>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f8354035000 nid=0x4049 runnable [0x00007f821b3f2000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051a045c28> (a sun.nio.ch.Util$2)
>> - locked <0x000000051a045c40> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051a000ce0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f834801a000 nid=0x4048 runnable [0x00007f821b4f3000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051a17bd60> (a sun.nio.ch.Util$2)
>> - locked <0x000000051a17bd78> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051a180230> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b374800
>> nid=0x4047 waiting on condition [0x00007f821b5f4000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051a08ba88> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857338a800
>> nid=0x4046 waiting on condition [0x00007f821b6f5000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051a147940> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>> tid=0x00007f85f3bff800 nid=0x4045 waiting on condition [0x00007f821b7f6000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000056dca65c0> (a
>> java.util.concurrent.FutureTask$Sync)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
>>  at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
>> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:227)
>>  at java.util.concurrent.FutureTask.get(FutureTask.java:91)
>> at
>> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:323)
>>  at
>> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:269)
>> at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:352)
>>  at
>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f8344035000 nid=0x4044 runnable [0x00007f821b8f7000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051a184a90> (a sun.nio.ch.Util$2)
>> - locked <0x000000051a184aa8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051a0bb2e0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bfd800
>> nid=0x4043 waiting on condition [0x00007f821b9f8000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051a147958> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f833001a000 nid=0x4042 runnable [0x00007f821baf9000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051a188f28> (a sun.nio.ch.Util$2)
>> - locked <0x000000051a188f40> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519d90758> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f8334035000 nid=0x4041 runnable [0x00007f821bbfa000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051a045cd0> (a sun.nio.ch.Util$2)
>> - locked <0x000000051a045ce8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051a000d28> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b372800
>> nid=0x4040 waiting on condition [0x00007f821bcfb000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519f74590> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573388800
>> nid=0x403f waiting on condition [0x00007f821bdfc000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519fb0c90> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f832c035000 nid=0x403e runnable [0x00007f821befd000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051a045d30> (a sun.nio.ch.Util$2)
>> - locked <0x000000051a045d48> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051a000d70> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bfb800
>> nid=0x403d waiting on condition [0x00007f821bffe000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519e5d388> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f831801a000 nid=0x403c runnable [0x00007f82341c0000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519ebdb70> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519ebdb88> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519d3a778> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f831001a000 nid=0x403b runnable [0x00007f82342c1000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519f47f90> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519f47fa8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519e11ca8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaf0000
>> nid=0x403a waiting on condition [0x00007f82343c2000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519f73018> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f8308035000 nid=0x4039 runnable [0x00007f82344c3000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519f745a8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519f745c0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519ca9f78> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573386800
>> nid=0x4038 waiting on condition [0x00007f82345c4000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519fa5ce0> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bf9800
>> nid=0x4037 waiting on condition [0x00007f82346c5000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519fb0ca8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f8300035000 nid=0x4036 runnable [0x00007f82347c6000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519f22d20> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519f22d38> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519d9f4b8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaee000
>> nid=0x4035 waiting on condition [0x00007f82348c7000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519fb0cc0> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f82fc01a000 nid=0x4034 runnable [0x00007f82349c8000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519f74608> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519f74620> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519ca9fc0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f82f001a000 nid=0x4033 runnable [0x00007f8234ac9000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519fb0cd8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519fb0cf0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519cadd40> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f44800
>> nid=0x4032 waiting on condition [0x00007f8234bca000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519ebdbd0> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3377000
>> nid=0x4031 waiting on condition [0x00007f8234ccb000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519f701c0> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f82e8035000 nid=0x4030 runnable [0x00007f8234dcc000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519f47ff0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519f48008> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519e11cf0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaec000
>> nid=0x402f waiting on condition [0x00007f8234ecd000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519fa5cf8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>> tid=0x00007f857fc38000 nid=0x402e waiting for monitor entry
>> [0x00007f8234fce000]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>> - waiting to lock <0x0000000518ee4c90> (a
>> org.apache.flume.channel.file.FlumeEventQueue)
>>  at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>> at
>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>  at
>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>> at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>  at
>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f82dc01a000 nid=0x402d runnable [0x00007f82350cf000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519ebdbe8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519ebdc00> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519d3a7c0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f82d001a000 nid=0x402c runnable [0x00007f82351d0000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519fb0d38> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519fb0d50> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519cadd88> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-144-thread-1" prio=10 tid=0x00007f85d8001000 nid=0x402b runnable
>> [0x00007f82352d1000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519f481a0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519f49040> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519e11d80> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6d3800
>> nid=0x402a waiting on condition [0x00007f82353d2000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519f22d80> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f42800
>> nid=0x4029 waiting on condition [0x00007f82354d3000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519fa5d10> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f82cc035000 nid=0x4028 runnable [0x00007f82355d4000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519fb0d98> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519fb0db0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519caddd0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-3" prio=10 tid=0x00007f8588001000 nid=0x4027 runnable
>> [0x00007f82356d5000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519ebdc48> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519ebdc60> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519d3a808> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3375000
>> nid=0x4026 waiting on condition [0x00007f82357d6000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519fa5d28> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f82c4035000 nid=0x4025 runnable [0x00007f82358d7000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519f22d98> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519f22db0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519d9f500> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaea000
>> nid=0x4024 waiting on condition [0x00007f82359d8000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519fa5d40> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f82a8035000 nid=0x4023 runnable [0x00007f8235ad9000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519fb0df8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519fb0e10> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519cade18> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f82b401a000 nid=0x4022 runnable [0x00007f8235bda000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519f73030> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519f73048> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519e69de8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f82ac035000 nid=0x4021 runnable [0x00007f8235cdb000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519f61380> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519f61398> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005197600f0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6d1800
>> nid=0x4020 waiting on condition [0x00007f8235ddc000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519f22df8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f40800
>> nid=0x401f waiting on condition [0x00007f8235edd000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519fa5d58> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3373000
>> nid=0x401e waiting on condition [0x00007f8235fde000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519ebdca8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f82a4035000 nid=0x401d runnable [0x00007f82360df000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519fb0e58> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519fb0e70> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519cade60> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a8000
>> nid=0x401c waiting on condition [0x00007f82361e0000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519f22e10> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-6" prio=10 tid=0x00007f85800a3800 nid=0x401b runnable
>> [0x00007f82362e1000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519ebde28> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519de7350> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519d3a898> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f828801a000 nid=0x401a runnable [0x00007f82363e2000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519fa5d70> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519fa5d88> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519d36338> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f828c035000 nid=0x4019 runnable [0x00007f82364e3000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519f73090> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519f730a8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519e69e30> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f8280035000 nid=0x4018 runnable [0x00007f82365e4000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519fb0eb8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519e1a7f8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519cadea8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6cf800
>> nid=0x4017 waiting on condition [0x00007f82366e5000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519f22e28> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f3e800
>> nid=0x4016 waiting on condition [0x00007f82367e6000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519f701d8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3371000
>> nid=0x4015 waiting on condition [0x00007f82368e7000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519fb0f00> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f8278035000 nid=0x4014 runnable [0x00007f82369e8000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519e93628> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519e93640> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519d9f548> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a6000
>> nid=0x4013 waiting on condition [0x00007f8236ae9000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005199bdaf0> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f827001a000 nid=0x4012 runnable [0x00007f8236bea000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519fb0f18> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519fb0f30> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519cadef0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6cd800
>> nid=0x4011 waiting on condition [0x00007f8236ceb000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519c47f38> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f826c035000 nid=0x4010 runnable [0x00007f8236dec000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519fa5dd0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519fa5de8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519d36380> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f826001a000 nid=0x400f runnable [0x00007f8236eed000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519c8c170> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519c8c188> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519a73d78> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572aeb000
>> nid=0x400e waiting on condition [0x00007f8236fee000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519c7e570> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2e800
>> nid=0x400d waiting on condition [0x00007f82370ef000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519c47f50> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f8258035000 nid=0x400b runnable [0x00007f82371f0000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519c4b190> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519c4b1a8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005199eb068> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a4000
>> nid=0x400a waiting on condition [0x00007f82372f1000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519c47f68> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f825001a000 nid=0x4009 runnable [0x00007f82373f2000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519c33550> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519c33568> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051996b7b8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e29c000
>> nid=0x4008 waiting on condition [0x00007f82374f3000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519c47f80> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f824c03f800 nid=0x4007 runnable [0x00007f82375f4000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519c7e588> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519c7e5a0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005199a1910> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f8240035000 nid=0x4006 runnable [0x00007f82376f5000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519c47f98> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519c47fb0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005199536b8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae9000
>> nid=0x4005 waiting on condition [0x00007f82377f6000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519c368c8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2c800
>> nid=0x4004 waiting on condition [0x00007f82378f7000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519c29bf0> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f8238035000 nid=0x4003 runnable [0x00007f82379f8000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519bd8198> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519bd81b0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519b59da0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a2000
>> nid=0x4002 waiting on condition [0x00007f8237af9000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519c7e5e8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f8230035000 nid=0x4001 runnable [0x00007f8237bfa000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519c47ff8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519c48010> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519953700> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e29a000
>> nid=0x4000 waiting on condition [0x00007f8237cfb000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519c7e600> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f823001a000 nid=0x3fff runnable [0x00007f8237dfc000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519c7e618> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519c7e630> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005199a1958> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f823c01a000 nid=0x3ffe runnable [0x00007f8237efd000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519c48058> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519c48070> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519a6d798> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f823801a000 nid=0x3ffd runnable [0x00007f8237ffe000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519c335b0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519c335c8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051996b800> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae7000
>> nid=0x3ffc waiting on condition [0x00007f84001c0000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519c480b8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2a800
>> nid=0x3ffb waiting on condition [0x00007f84002c1000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519bd81f8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24e800
>> nid=0x3ffa waiting on condition [0x00007f84003c2000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519c29c08> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f824001a000 nid=0x3ff9 runnable [0x00007f84004c3000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519c7e678> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519c7e690> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005199a19a0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e298000
>> nid=0x3ff8 waiting on condition [0x00007f84005c4000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519c67408> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-5" prio=10 tid=0x00007f8580088800 nid=0x3ff7 runnable
>> [0x00007f84006c5000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519c48238> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519d5e0d0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519a6d828> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f825c01a000 nid=0x3ff6 runnable [0x00007f84007c6000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519c368e0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519c368f8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519953088> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f825801a000 nid=0x3ff5 runnable [0x00007f84008c7000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519c33610> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519c33628> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051996b848> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f826401a000 nid=0x3ff4 runnable [0x00007f84009c8000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519c7e6d8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519c7e6f0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005199a19e8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae5000
>> nid=0x3ff3 waiting on condition [0x00007f8400ac9000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519c730b8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24c800
>> nid=0x3ff2 waiting on condition [0x00007f8400bca000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519c29c20> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f29000
>> nid=0x3ff1 waiting on condition [0x00007f8400ccb000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519c4b1f0> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f826c01a000 nid=0x3ff0 runnable [0x00007f8400dcc000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519bd8210> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519bd8228> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519b59de8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e296800
>> nid=0x3fef waiting on condition [0x00007f8400ecd000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519c8c1d0> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f827801a000 nid=0x3fee runnable [0x00007f8400fce000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519c7e738> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519c7e750> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005199a1a30> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f828401a000 nid=0x3fed runnable [0x00007f84010cf000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519c33670> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519c33688> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051996b890> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f828001a000 nid=0x3fec runnable [0x00007f84011d0000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519c29c38> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519c29c50> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519955c68> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f828c01a000 nid=0x3feb runnable [0x00007f84012d1000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519c7e798> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519c7e7b0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005199533a0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae6000
>> nid=0x3fea waiting on condition [0x00007f84013d2000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519bd8270> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24a800
>> nid=0x3fe9 waiting on condition [0x00007f84014d3000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519c36940> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572273800
>> nid=0x3fe8 waiting on condition [0x00007f84015d4000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519c8c1e8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de54800
>> nid=0x3fe7 waiting on condition [0x00007f84016d5000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519c67420> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f829001a000 nid=0x3fe6 runnable [0x00007f84017d6000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519c7e7f8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519c7e810> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005199533e8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f82a401a000 nid=0x3fe5 runnable [0x00007f84018d7000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519c336d0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519c336e8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051996b8d8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae4000
>> nid=0x3fe4 waiting on condition [0x00007f84019d8000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519946350> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-4" prio=10 tid=0x00007f858006d800 nid=0x3fe3 runnable
>> [0x00007f8401ad9000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519bd83d8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519f042a0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519b59e78> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f82ac01a000 nid=0x3fe2 runnable [0x00007f8401bda000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519c7e858> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519c7e870> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519953430> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a248800
>> nid=0x3fe1 waiting on condition [0x00007f8401cdb000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519890ec8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f82a801a000 nid=0x3fe0 runnable [0x00007f8401ddc000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519c8c200> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519c8c218> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519a73dc0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572271800
>> nid=0x3fdf waiting on condition [0x00007f8401edd000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005198a3170> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de52800
>> nid=0x3fde waiting on condition [0x00007f8401fde000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005198b7610> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f82b801a000 nid=0x3fdd runnable [0x00007f84020df000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519947f70> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519947f88> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005197bc938> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f82c401a000 nid=0x3fdc runnable [0x00007f84021e0000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519946368> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519946380> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051983b9b8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f82cc01a000 nid=0x3fdb runnable [0x00007f84022e1000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005198a3188> (a sun.nio.ch.Util$2)
>> - locked <0x00000005198a31a0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005197300e0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae2000
>> nid=0x3fda waiting on condition [0x00007f84023e2000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005199463c8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857226f800
>> nid=0x3fd9 waiting on condition [0x00007f84024e3000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519792380> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f82c801a000 nid=0x3fd8 runnable [0x00007f84025e4000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519947fd0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519947fe8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005197bc980> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579597000
>> nid=0x3fd7 waiting on condition [0x00007f84026e5000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519488cc0> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de50800
>> nid=0x3fd6 waiting on condition [0x00007f84027e6000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005198cc640> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f82d801a000 nid=0x3fd5 runnable [0x00007f84028e7000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005199463e0> (a sun.nio.ch.Util$2)
>> - locked <0x00000005199463f8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051983ba00> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f82e401a000 nid=0x3fd4 runnable [0x00007f84029e8000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005198f7fc8> (a sun.nio.ch.Util$2)
>> - locked <0x00000005198f7fe0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005194d0160> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f82ec01a000 nid=0x3fd3 runnable [0x00007f8402ae9000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005198a31e8> (a sun.nio.ch.Util$2)
>> - locked <0x00000005198a3200> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519730128> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae0800
>> nid=0x3fd2 waiting on condition [0x00007f8402bea000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519948030> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857226d800
>> nid=0x3fd1 waiting on condition [0x00007f8402ceb000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519946440> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f82e801a000 nid=0x3fd0 runnable [0x00007f8402dec000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519890ee0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519890ef8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005196f56f8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579595000
>> nid=0x3fcf waiting on condition [0x00007f8402eed000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005198b7628> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de4e800
>> nid=0x3fce waiting on condition [0x00007f8402fee000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005198a3248> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f830401a000 nid=0x3fcd runnable [0x00007f84030ef000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519946458> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519946470> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051983ba48> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f830001a000 nid=0x3fcc runnable [0x00007f84031f0000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519937b88> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519937ba0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519887cb8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f830c01a000 nid=0x3fcb runnable [0x00007f84032f1000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519948048> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519948060> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005197bc9c8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f830801a000 nid=0x3fca runnable [0x00007f84033f2000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005199464b8> (a sun.nio.ch.Util$2)
>> - locked <0x00000005199464d0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051983ba90> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e2b000
>> nid=0x3fc9 waiting on condition [0x00007f84034f3000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519943980> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1595000
>> nid=0x3fc8 waiting on condition [0x00007f84035f4000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005198a3260> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857bff5800
>> nid=0x3fc7 waiting on condition [0x00007f84036f5000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519901138> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da0c000
>> nid=0x3fc6 waiting on condition [0x00007f84037f6000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005199480a8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f832001a000 nid=0x3fc5 runnable [0x00007f84038f7000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519946518> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519946530> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051983bad8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f832c01a000 nid=0x3fc4 runnable [0x00007f84039f8000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005198a3278> (a sun.nio.ch.Util$2)
>> - locked <0x00000005198a3290> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519730170> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f832801a000 nid=0x3fc3 runnable [0x00007f8403af9000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005198f8028> (a sun.nio.ch.Util$2)
>> - locked <0x00000005198f8040> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005194d01a8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f833401a000 nid=0x3fc2 runnable [0x00007f8403bfa000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005198b7640> (a sun.nio.ch.Util$2)
>> - locked <0x00000005198b7658> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005196f3480> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e29000
>> nid=0x3fc1 waiting on condition [0x00007f8403cfb000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519946578> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1593000
>> nid=0x3fc0 waiting on condition [0x00007f8403dfc000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005199480c0> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579592000
>> nid=0x3fbf waiting on condition [0x00007f8403efd000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519943998> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da0a800
>> nid=0x3fbe waiting on condition [0x00007f84401c0000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519946590> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f834401a000 nid=0x3fbd runnable [0x00007f84402c1000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519903410> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519903428> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005191a1498> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f834001a000 nid=0x3fbc runnable [0x00007f84403c2000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519943ef0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519943f08> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005196f5740> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f834c01a000 nid=0x3fbb runnable [0x00007f84404c3000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005199497b0> (a sun.nio.ch.Util$2)
>> - locked <0x00000005199497c8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519685f60> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573ff1800
>> nid=0x3fba waiting on condition [0x00007f84405c4000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005198f8088> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1591800
>> nid=0x3fb9 waiting on condition [0x00007f84406c5000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005198a32d8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579591000
>> nid=0x3fb8 waiting on condition [0x00007f84407c6000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005199480d8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f835401a000 nid=0x3fb7 runnable [0x00007f84408c7000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005199465a8> (a sun.nio.ch.Util$2)
>> - locked <0x00000005199465c0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051983bb20> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da08800
>> nid=0x3fb6 waiting on condition [0x00007f84409c8000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005199439b0> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-142-thread-1" prio=10 tid=0x00007f8584001000 nid=0x3fb5 runnable
>> [0x00007f8440ac9000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005198b77f0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519a00960> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519952e48> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f837401a000 nid=0x3fb4 runnable [0x00007f8440bca000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519937be8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519937c00> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519887d00> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f836801a000 nid=0x3fb3 runnable [0x00007f8440ccb000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519943f50> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519943f68> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005196f5788> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f837001a000 nid=0x3fb2 runnable [0x00007f8440dcc000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519901150> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519901168> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519106100> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f158f800
>> nid=0x3fb1 waiting on condition [0x00007f8440ecd000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519946608> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e26800
>> nid=0x3fb0 waiting on condition [0x00007f8440fce000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005198a32f0> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913e000
>> nid=0x3faf waiting on condition [0x00007f84410cf000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005199439c8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f837801a000 nid=0x3fae runnable [0x00007f84411d0000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005199480f0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519948108> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005197bca10> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da07000
>> nid=0x3fad waiting on condition [0x00007f84412d1000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519943fb0> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f838801a000 nid=0x3fac runnable [0x00007f84413d2000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005198f80a0> (a sun.nio.ch.Util$2)
>> - locked <0x00000005198f80b8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005194d01f0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f839401a000 nid=0x3fab runnable [0x00007f84414d3000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519946620> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519946638> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051983bb68> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f839001a000 nid=0x3faa runnable [0x00007f84415d4000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519937c48> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519937c60> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519887d48> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d0d000
>> nid=0x3fa9 waiting on condition [0x00007f84416d5000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519654310> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913c000
>> nid=0x3fa8 waiting on condition [0x00007f84417d6000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005191b54d8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571164000
>> nid=0x3fa7 waiting on condition [0x00007f84418d7000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519637a20> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f83a801a000 nid=0x3fa6 runnable [0x00007f84419d8000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005195f76c8> (a sun.nio.ch.Util$2)
>> - locked <0x00000005196430c0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005194ecaf0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f83b401a000 nid=0x3fa5 runnable [0x00007f8441ad9000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051965f960> (a sun.nio.ch.Util$2)
>> - locked <0x000000051965f978> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519104648> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f83b001a000 nid=0x3fa4 runnable [0x00007f8441bda000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519655e28> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519655e40> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519104ba0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-3" prio=10 tid=0x00007f8580052800 nid=0x3fa3 runnable
>> [0x00007f8441cdb000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519637b10> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519694210> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051916e518> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f83bc01a000 nid=0x3fa2 runnable [0x00007f8441ddc000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519597678> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519597690> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519106388> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c4800
>> nid=0x3fa1 waiting on condition [0x00007f8441edd000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519654328> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d0b000
>> nid=0x3fa0 waiting on condition [0x00007f8441fde000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519676ad0> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571162000
>> nid=0x3f9f waiting on condition [0x00007f84420df000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519655618> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913a000
>> nid=0x3f9e waiting on condition [0x00007f84421e0000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005194ae248> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-2" prio=10 tid=0x00007f8580001000 nid=0x3f9d runnable
>> [0x00007f84422e1000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519654340> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519654358> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519105380> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f83cc01a000 nid=0x3f9c runnable [0x00007f84423e2000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519655e88> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519655ea0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519104be8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f83d001a000 nid=0x3f9b runnable [0x00007f84424e3000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005194e1350> (a sun.nio.ch.Util$2)
>> - locked <0x00000005194e1368> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005191041c8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c3000
>> nid=0x3f9a waiting on condition [0x00007f84425e4000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051967caf8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f83d801a000 nid=0x3f99 runnable [0x00007f84426e5000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519655ee8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519655f00> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519104c30> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f83dc01a000 nid=0x3f98 runnable [0x00007f84427e6000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005196543a0> (a sun.nio.ch.Util$2)
>> - locked <0x00000005196543b8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005191053c8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d09800
>> nid=0x3f97 waiting on condition [0x00007f84428e7000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519655630> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579138000
>> nid=0x3f96 waiting on condition [0x00007f84429e8000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005196592c8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571160000
>> nid=0x3f95 waiting on condition [0x00007f8442ae9000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519655f48> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f840801a000 nid=0x3f94 runnable [0x00007f8442bea000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519655648> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519655660> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051914b290> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f83e801a000 nid=0x3f93 runnable [0x00007f8442ceb000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519643108> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519643120> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005194ecb38> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d07800
>> nid=0x3f92 waiting on condition [0x00007f8442dec000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519658428> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f83f801a000 nid=0x3f91 runnable [0x00007f8442eed000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005194c9908> (a sun.nio.ch.Util$2)
>> - locked <0x00000005194c9920> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519511968> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f83fc01a000 nid=0x3f90 runnable [0x00007f8442fee000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519643168> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519643180> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005194ecb80> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578465000
>> nid=0x3f8f waiting on condition [0x00007f8403ffe000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519654400> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857115e800
>> nid=0x3f8e waiting on condition [0x00007f84430ef000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005196556a8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c1000
>> nid=0x3f8d waiting on condition [0x00007f84431f0000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519654418> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f840c01a000 nid=0x3f8c runnable [0x00007f84432f1000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005196431c8> (a sun.nio.ch.Util$2)
>> - locked <0x00000005196431e0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005194ecbc8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f841801a000 nid=0x3f8b runnable [0x00007f84433f2000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519655f60> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519655f78> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519104c78> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b4000
>> nid=0x3f8a waiting on condition [0x00007f84434f3000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005196721e8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5bf000
>> nid=0x3f89 waiting on condition [0x00007f84435f4000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519654430> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f841c01a000 nid=0x3f88 runnable [0x00007f84436f5000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005196556c0> (a sun.nio.ch.Util$2)
>> - locked <0x00000005196556d8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051914b2d8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f842801a000 nid=0x3f87 runnable [0x00007f84437f6000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519643228> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519643240> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005194ecc10> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b2000
>> nid=0x3f86 waiting on condition [0x00007f84438f7000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519654448> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5bd800
>> nid=0x3f85 waiting on condition [0x00007f84439f8000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051967cb10> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f843001a000 nid=0x3f84 runnable [0x00007f8443af9000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519655fc0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519655fd8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519104cc0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b0800
>> nid=0x3f83 waiting on condition [0x00007f8443bfa000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005196592e0> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f843801a000 nid=0x3f82 runnable [0x00007f8443cfb000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519643288> (a sun.nio.ch.Util$2)
>> - locked <0x00000005196432a0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005194ecc58> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d17b000
>> nid=0x3f81 waiting on condition [0x00007f8443dfc000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051921afd8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f843c01a000 nid=0x3f80 runnable [0x00007f8443efd000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519654460> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519654478> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519105410> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08ae800
>> nid=0x3f7f waiting on condition [0x00007f8443ffe000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519676ae8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f844401a000 nid=0x3f7e runnable [0x00007f85441a6000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051967cb28> (a sun.nio.ch.Util$2)
>> - locked <0x000000051967cb40> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005194745e0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d179000
>> nid=0x3f7d waiting on condition [0x00007f85442a7000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005196432e8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f845801a000 nid=0x3f7c runnable [0x00007f85443a8000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519658440> (a sun.nio.ch.Util$2)
>> - locked <0x00000005196583f8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519349128> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f845401a000 nid=0x3f7b runnable [0x00007f85444a9000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519676aa0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519676ab8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051921b020> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08ad800
>> nid=0x3f7a waiting on condition [0x00007f85446c5000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519659298> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578463000
>> nid=0x3f79 waiting on condition [0x00007f85447c6000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005194a3568> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f845c01a000 nid=0x3f78 runnable [0x00007f85445aa000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005196544c0> (a sun.nio.ch.Util$2)
>> - locked <0x00000005196544d8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519105458> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d177000
>> nid=0x3f77 waiting on condition [0x00007f85448c7000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519656020> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f846401a000 nid=0x3f76 runnable [0x00007f85449c8000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519655720> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519655738> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519105f98> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857115c800
>> nid=0x3f75 waiting on condition [0x00007f8544ac9000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519402f58> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f847801a000 nid=0x3f74 runnable [0x00007f8544bca000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519654520> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519654538> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005191054a0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f847401a000 nid=0x3f73 runnable [0x00007f8544ccb000]
>>    java.lang.Thread.State: RUNNABLE
>> at java.util.HashMap.newKeyIterator(HashMap.java:840)
>> at java.util.HashMap$KeySet.iterator(HashMap.java:874)
>>  at java.util.HashSet.iterator(HashSet.java:153)
>> at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
>>  - locked <0x0000000519656248> (a java.util.HashSet)
>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)
>>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>> - locked <0x0000000519656038> (a sun.nio.ch.Util$2)
>>  - locked <0x0000000519656050> (a java.util.Collections$UnmodifiableSet)
>> - locked <0x0000000519104d08> (a sun.nio.ch.EPollSelectorImpl)
>>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>> at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>  at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>  at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0015800
>> nid=0x3f72 waiting on condition [0x00007f8544dcc000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>  at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051941d010> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578461000
>> nid=0x3f71 waiting on condition [0x00007f8544ecd000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519477938> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f847c01a000 nid=0x3f70 runnable [0x00007f8544fce000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051942aa08> (a sun.nio.ch.Util$2)
>> - locked <0x000000051942aa20> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519349050> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d175800
>> nid=0x3f6f waiting on condition [0x00007f85450cf000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519470c60> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f848401a000 nid=0x3f6e runnable [0x00007f85451d0000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519445fa0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519445fb8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005194679d0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570456800
>> nid=0x3f6d waiting on condition [0x00007f85452d1000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519419230> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f849801a000 nid=0x3f6c runnable [0x00007f85453d2000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519470c78> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519470c90> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005194743a0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f849401a000 nid=0x3f6b runnable [0x00007f85454d3000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051945beb8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051945bed0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519196000> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0013800
>> nid=0x3f6a waiting on condition [0x00007f85455d4000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519470cd8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857845f000
>> nid=0x3f69 waiting on condition [0x00007f85456d5000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051941d1c0> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f849c01a000 nid=0x3f68 runnable [0x00007f85457d6000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005194583b8> (a sun.nio.ch.Util$2)
>> - locked <0x00000005194583d0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005194743e8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c488000
>> nid=0x3f67 waiting on condition [0x00007f85458d7000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519424e80> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f84a401a000 nid=0x3f66 runnable [0x00007f85459d8000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051941d1d8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051941d1f0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519467a18> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570455000
>> nid=0x3f65 waiting on condition [0x00007f8545ad9000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051945bf18> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f84b801a000 nid=0x3f64 runnable [0x00007f8545bda000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519419248> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519419200> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519431ee0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f84c001a000 nid=0x3f63 runnable [0x00007f8545cdb000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051941d238> (a sun.nio.ch.Util$2)
>> - locked <0x000000051941d250> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519467a60> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0012000
>> nid=0x3f62 waiting on condition [0x00007f8545ddc000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519458418> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f84bc01a000 nid=0x3f61 runnable [0x00007f8545edd000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051945bf30> (a sun.nio.ch.Util$2)
>> - locked <0x000000051945bf48> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519196048> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857845e000
>> nid=0x3f60 waiting on condition [0x00007f8545fde000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519479f10> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c486000
>> nid=0x3f5f waiting on condition [0x00007f85460df000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051941d298> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f84c401a000 nid=0x3f5e runnable [0x00007f85461e0000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051942a9a8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051942a9c0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519349098> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570453800
>> nid=0x3f5d waiting on condition [0x00007f85462e1000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051945bf90> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f84cc01a000 nid=0x3f5c runnable [0x00007f85463e2000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519458430> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519458448> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519474430> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c484800
>> nid=0x3f5b waiting on condition [0x00007f85464e3000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051941d2b0> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f84d401a000 nid=0x3f5a runnable [0x00007f85465e4000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005194191e8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051936a4e0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519431f28> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570450800
>> nid=0x3f59 waiting on condition [0x00007f85466e5000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051943ef60> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f84dc01a000 nid=0x3f58 runnable [0x00007f85467e6000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051942a948> (a sun.nio.ch.Util$2)
>> - locked <0x000000051942a960> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005193490e0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c482800
>> nid=0x3f57 waiting on condition [0x00007f85468e7000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051945bfa8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f84e401a000 nid=0x3f56 runnable [0x00007f85469e8000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051941d2c8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051941d2e0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519467aa8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857044f000
>> nid=0x3f55 waiting on condition [0x00007f8546ae9000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051945bfc0> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f84f801a000 nid=0x3f54 runnable [0x00007f8546bea000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051936a4c8> (a sun.nio.ch.Util$2)
>> - locked <0x000000051936a480> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519431f70> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f84f401a000 nid=0x3f53 runnable [0x00007f8546ceb000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519458490> (a sun.nio.ch.Util$2)
>> - locked <0x00000005194584a8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519474478> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578008000
>> nid=0x3f52 waiting on condition [0x00007f8546dec000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051942a8e8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c00d000
>> nid=0x3f51 waiting on condition [0x00007f8546eed000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000519447098> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f84fc01a000 nid=0x3f50 runnable [0x00007f8546fee000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051941d328> (a sun.nio.ch.Util$2)
>> - locked <0x000000051941d340> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519467af0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857044d000
>> nid=0x3f4f waiting on condition [0x00007f85470ef000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051945bfd8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f851001a000 nid=0x3f4e runnable [0x00007f85471f0000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519424e98> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519424e50> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519380d20> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f850c01a000 nid=0x3f4d runnable [0x00007f85472f1000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005194584f0> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519458508> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005194744c0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578006800
>> nid=0x3f4c waiting on condition [0x00007f85473f2000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051945bff0> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c01d800
>> nid=0x3f4b waiting on condition [0x00007f85474f3000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051936a468> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f851401a000 nid=0x3f4a runnable [0x00007f85475f4000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051941d388> (a sun.nio.ch.Util$2)
>> - locked <0x000000051941d3a0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519467b38> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857045b800
>> nid=0x3f49 waiting on condition [0x00007f85476f5000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051942a900> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f853801a000 nid=0x3f48 runnable [0x00007f85477f6000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519479f28> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519479ee0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519203068> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f853001f800 nid=0x3f47 runnable [0x00007f85478f7000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519477950> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519477968> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051922faf0> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f852c01f000 nid=0x3f46 runnable [0x00007f85479f8000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005194353f8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519435410> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x000000051921b0f8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver I/O Worker 1" daemon prio=10
>> tid=0x00007f853401a000 nid=0x3f45 runnable [0x00007f8547af9000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051945c008> (a sun.nio.ch.Util$2)
>> - locked <0x000000051945c020> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000519196090> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578005800
>> nid=0x3f44 waiting on condition [0x00007f8547bfa000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005190de788> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f001a800
>> nid=0x3f43 waiting on condition [0x00007f8547cfb000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000518fae040> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c022000
>> nid=0x3f42 waiting on condition [0x00007f8547dfc000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000518cafb20> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570460000
>> nid=0x3f41 waiting on condition [0x00007f8547efd000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005190db9c8> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>  at
>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-2" prio=10 tid=0x00007f8588036000 nid=0x3f40 runnable
>> [0x00007f8547ffe000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005190ee0e8> (a sun.nio.ch.Util$2)
>> - locked <0x00000005191f0be8> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005190e1d18> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-140-thread-1" prio=10 tid=0x00007f858801b800 nid=0x3f3f runnable
>> [0x00007f858c18a000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000518f16658> (a sun.nio.ch.Util$2)
>> - locked <0x000000051941f820> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005190edc80> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-139-thread-1" prio=10 tid=0x00007f8580002800 nid=0x3f3e runnable
>> [0x00007f858c28b000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x000000051913eb50> (a sun.nio.ch.Util$2)
>> - locked <0x000000051913eca0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000518c9c4e8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-148-thread-1" prio=10 tid=0x00007f856844c000 nid=0x3f3d waiting on
>> condition [0x00007f858c38c000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x00000005190cfe98> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-146-thread-1" prio=10 tid=0x00007f85dc006800 nid=0x3f3c waiting on
>> condition [0x00007f858c48d000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>>  - parking to wait for  <0x0000000519018558> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>  at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-147-thread-1" prio=10 tid=0x00007f85e82b1000 nid=0x3f3b waiting on
>> condition [0x00007f858c58e000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051903b488> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-145-thread-1" prio=10 tid=0x00007f8574027000 nid=0x3f3a waiting on
>> condition [0x00007f858c68f000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>>  - parking to wait for  <0x000000051903b998> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>  at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-141-thread-1" prio=10 tid=0x00007f85e8159000 nid=0x3f39 runnable
>> [0x00007f861cd8e000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005190db9e0> (a sun.nio.ch.Util$2)
>> - locked <0x00000005190db998> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000518c9c020> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-138-thread-1" prio=10 tid=0x00007f856844a800 nid=0x3f38 runnable
>> [0x00007f861cf90000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x00000005190e1b68> (a sun.nio.ch.Util$2)
>> - locked <0x00000005190e1b80> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x0000000518c9c608> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-143-thread-1" prio=10 tid=0x00007f85dc003000 nid=0x3f37 runnable
>> [0x00007f861d091000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519018fa8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519018fc0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005190e2228> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "pool-137-thread-1" prio=10 tid=0x00007f8574007000 nid=0x3f36 runnable
>> [0x00007f861ce8f000]
>>    java.lang.Thread.State: RUNNABLE
>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>  - locked <0x0000000519003dc8> (a sun.nio.ch.Util$2)
>> - locked <0x0000000519003de0> (a java.util.Collections$UnmodifiableSet)
>>  - locked <0x00000005190ea1a8> (a sun.nio.ch.EPollSelectorImpl)
>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>  at
>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
>> at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "1407844613@qtp-1263282395-1 - Acceptor0 SocketConnector@0.0.0.0:54321"
>> prio=10 tid=0x00007f85e404a800 nid=0x3f35 runnable [0x00007f858c790000]
>>    java.lang.Thread.State: RUNNABLE
>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
>>  - locked <0x00000005190e2720> (a java.net.SocksSocketImpl)
>> at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>>  at java.net.ServerSocket.accept(ServerSocket.java:430)
>> at org.mortbay.jetty.bio.SocketConnector.accept(SocketConnector.java:99)
>>  at
>> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
>> at
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>
>> "208924888@qtp-1263282395-0" prio=10 tid=0x00007f85e4021800 nid=0x3f34
>> in Object.wait() [0x00007f861c15b000]
>>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>  at java.lang.Object.wait(Native Method)
>>  - waiting on <0x00000005190d0528> (a
>> org.mortbay.thread.QueuedThreadPool$PoolThread)
>> at
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
>>  - locked <0x00000005190d0528> (a
>> org.mortbay.thread.QueuedThreadPool$PoolThread)
>>
>> "lifecycleSupervisor-1-9" prio=10 tid=0x00007f85e401a800 nid=0x3f33
>> waiting on condition [0x00007f861c687000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000518c85670> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "lifecycleSupervisor-1-8" prio=10 tid=0x00007f85e4018800 nid=0x3f32
>> waiting on condition [0x00007f861c788000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>>  - parking to wait for  <0x0000000518c85670> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>  at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "lifecycleSupervisor-1-7" prio=10 tid=0x00007f85e4016800 nid=0x3f31
>> waiting on condition [0x00007f861c889000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000518c85670> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "lifecycleSupervisor-1-6" prio=10 tid=0x00007f85e4014800 nid=0x3f30
>> waiting on condition [0x00007f861c98a000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>>  - parking to wait for  <0x0000000518c85670> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>  at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "lifecycleSupervisor-1-5" prio=10 tid=0x00007f85e4013000 nid=0x3f2f
>> waiting on condition [0x00007f861ca8b000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000518c85670> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "lifecycleSupervisor-1-4" prio=10 tid=0x00007f85e4028000 nid=0x3f2e
>> waiting on condition [0x00007f861cb8c000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>>  - parking to wait for  <0x0000000518c85670> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>  at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "lifecycleSupervisor-1-3" prio=10 tid=0x00007f85e4026800 nid=0x3f2d
>> waiting on condition [0x00007f861cc8d000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000518c85670> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "Log-BackgroundWorker-ch1" prio=10 tid=0x00007f85dc001000 nid=0x3f27
>> waiting on condition [0x00007f861d299000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>>  - parking to wait for  <0x0000000519017718> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>  at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "Log-BackgroundWorker-ch2" prio=10 tid=0x00007f85e80ca000 nid=0x3f26
>> waiting on condition [0x00007f861d39a000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x000000051903b4e8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "lifecycleSupervisor-1-2" prio=10 tid=0x00007f85e4025000 nid=0x3f25
>> waiting on condition [0x00007f861d6ba000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>>  - parking to wait for  <0x0000000518c85670> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>  at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "lifecycleSupervisor-1-1" prio=10 tid=0x00007f85e400e800 nid=0x3f24
>> waiting on condition [0x00007f861d7bb000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000518c85670> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "conf-file-poller-0" prio=10 tid=0x00007f85f0001000 nid=0x3f23 waiting on
>> condition [0x00007f861d8bc000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>>  - parking to wait for  <0x0000000518ec72f8> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>  at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>>  at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>  at java.lang.Thread.run(Thread.java:662)
>>
>> "DestroyJavaVM" prio=10 tid=0x00007f86a800c800 nid=0x3efa waiting on
>> condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "lifecycleSupervisor-1-0" prio=10 tid=0x00007f86a89f5800 nid=0x3f22
>> waiting on condition [0x00007f861d9bd000]
>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>  at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for  <0x0000000518c85670> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>  at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>> at
>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> "Low Memory Detector" daemon prio=10 tid=0x00007f86a831d800 nid=0x3f20
>> runnable [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "C2 CompilerThread1" daemon prio=10 tid=0x00007f86a831b000 nid=0x3f1f
>> waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "C2 CompilerThread0" daemon prio=10 tid=0x00007f86a8319000 nid=0x3f1e
>> waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "Signal Dispatcher" daemon prio=10 tid=0x00007f86a8316800 nid=0x3f1d
>> runnable [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "Surrogate Locker Thread (Concurrent GC)" daemon prio=10
>> tid=0x00007f86a8314800 nid=0x3f1c waiting on condition [0x0000000000000000]
>>    java.lang.Thread.State: RUNNABLE
>>
>> "Finalizer" daemon prio=10 tid=0x00007f86a82f9800 nid=0x3f1b in
>> Object.wait() [0x00007f861e3eb000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>>  at java.lang.Object.wait(Native Method)
>> - waiting on <0x0000000518c843c0> (a java.lang.ref.ReferenceQueue$Lock)
>>  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>> - locked <0x0000000518c843c0> (a java.lang.ref.ReferenceQueue$Lock)
>>  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>
>> "Reference Handler" daemon prio=10 tid=0x00007f86a82f7800 nid=0x3f1a in
>> Object.wait() [0x00007f861e4ec000]
>>    java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>>  - waiting on <0x0000000518c80860> (a java.lang.ref.Reference$Lock)
>> at java.lang.Object.wait(Object.java:485)
>>  at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>> - locked <0x0000000518c80860> (a java.lang.ref.Reference$Lock)
>>
>> "VM Thread" prio=10 tid=0x00007f86a82f1000 nid=0x3f19 runnable
>>
>> "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f86a801b800
>> nid=0x3efb runnable
>>
>> "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f86a801d000
>> nid=0x3efc runnable
>>
>> "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f86a801f000
>> nid=0x3efd runnable
>>
>> "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f86a8021000
>> nid=0x3efe runnable
>>
>> "Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x00007f86a8022800
>> nid=0x3eff runnable
>>
>> "Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x00007f86a8024800
>> nid=0x3f00 runnable
>>
>> "Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x00007f86a8026000
>> nid=0x3f01 runnable
>>
>> "Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x00007f86a8028000
>> nid=0x3f02 runnable
>>
>> "Gang worker#8 (Parallel GC Threads)" prio=10 tid=0x00007f86a802a000
>> nid=0x3f03 runnable
>>
>> "Gang worker#9 (Parallel GC Threads)" prio=10 tid=0x00007f86a802b800
>> nid=0x3f04 runnable
>>
>> "Gang worker#10 (Parallel GC Threads)" prio=10 tid=0x00007f86a802d800
>> nid=0x3f05 runnable
>>
>> "Gang worker#11 (Parallel GC Threads)" prio=10 tid=0x00007f86a802f800
>> nid=0x3f06 runnable
>>
>> "Gang worker#12 (Parallel GC Threads)" prio=10 tid=0x00007f86a8031000
>> nid=0x3f07 runnable
>>
>> "Gang worker#13 (Parallel GC Threads)" prio=10 tid=0x00007f86a8033000
>> nid=0x3f08 runnable
>>
>> "Gang worker#14 (Parallel GC Threads)" prio=10 tid=0x00007f86a8035000
>> nid=0x3f09 runnable
>>
>> "Gang worker#15 (Parallel GC Threads)" prio=10 tid=0x00007f86a8036800
>> nid=0x3f0a runnable
>>
>> "Gang worker#16 (Parallel GC Threads)" prio=10 tid=0x00007f86a8038800
>> nid=0x3f0b runnable
>>
>> "Gang worker#17 (Parallel GC Threads)" prio=10 tid=0x00007f86a803a000
>> nid=0x3f0c runnable
>>
>> "Gang worker#18 (Parallel GC Threads)" prio=10 tid=0x00007f86a803c000
>> nid=0x3f0d runnable
>>
>> "Gang worker#19 (Parallel GC Threads)" prio=10 tid=0x00007f86a803e000
>> nid=0x3f0e runnable
>>
>> "Gang worker#20 (Parallel GC Threads)" prio=10 tid=0x00007f86a803f800
>> nid=0x3f0f runnable
>>
>> "Gang worker#21 (Parallel GC Threads)" prio=10 tid=0x00007f86a8041800
>> nid=0x3f10 runnable
>>
>> "Gang worker#22 (Parallel GC Threads)" prio=10 tid=0x00007f86a8043800
>> nid=0x3f11 runnable
>>
>> "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f86a82aa000
>> nid=0x3f18 runnable
>> "Gang worker#0 (Parallel CMS Threads)" prio=10 tid=0x00007f86a829e800
>> nid=0x3f12 runnable
>>
>> "Gang worker#1 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a0000
>> nid=0x3f13 runnable
>>
>> "Gang worker#2 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a2000
>> nid=0x3f14 runnable
>>
>> "Gang worker#3 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a3800
>> nid=0x3f15 runnable
>>
>> "Gang worker#4 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a5800
>> nid=0x3f16 runnable
>>
>> "Gang worker#5 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a7800
>> nid=0x3f17 runnable
>>
>> "VM Periodic Task Thread" prio=10 tid=0x00007f86a8329000 nid=0x3f21
>> waiting on condition
>>
>> JNI global references: 1749
>>
>>
>>
>> --
>>
>>
>> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 |
>> pankaj@brightroll.com
>>
>> Pankaj Gupta | Software Engineer
>>
>> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>>
>>
>> United States | Canada | United Kingdom | Germany
>>
>>
>> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
>> !
>>
>>
>>
>>
>>
>> --
>>
>>
>> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 |
>> pankaj@brightroll.com
>>
>> Pankaj Gupta | Software Engineer
>>
>> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>>
>>
>> United States | Canada | United Kingdom | Germany
>>
>>
>> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
>> !
>>
>>
>>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>



-- 


*P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com

Pankaj Gupta | Software Engineer

*BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com


United States | Canada | United Kingdom | Germany


We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
!

Re: Lock contention in FileChannel

[Brock Noland <br...@cloudera.com>]

The lock is per file. Adding more directories to the channel will cause
more files to be created. Of course you'll need additional disks behind
those directories to see any performance increase.


On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <hshreedharan@cloudera.com
> wrote:

>  Yes, IO is done inside locks to avoid multiple takes and puts getting
> written out at the same time. Even though Java makes sure the writes are
> serialized, Flume still needs to keep track of some counters etc, so the
> lock is required. Note that the lock you are talking about is  in the
> LogFile class, which represents a single file - so even if the write is
> inside that lock (which is also inside that class itself) that  does not
> cause any contention - because the lock is just preventing two IO ops to
> happen at the same time.
>
>
> Thanks,
> Hari
>
> On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:
>
> It seems like some i/o is done inside the lock, which means that time for
> taking a lock is proportional to the time for i/o and thus it becomes a
> problem. I apologize in advance if I am wrong but the call stack and
> behavior I'm seeing seems to suggest that. Specifically, it seems that we
> do a write while inside take:
> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.NativeThread.current(Native Method)
>         at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>         at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>         - locked <0x00000005190ec998> (a java.lang.Object)
>         at
> org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>         at
> org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>         - locked <0x0000000519111590> (a
> org.apache.flume.channel.file.LogFileV3$Writer)
>         at org.apache.flume.channel.file.Log.take(Log.java:550)
>         at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>         at
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>         at
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>         at
> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>         at
> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:662)
>
>
>
> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <
> hshreedharan@cloudera.com> wrote:
>
> Since the channel is designed to make sure that events are not duplicated
> to multiple sinks, and to protect against corruption due to concurrency
> issues, we do not need the locking in the channel's flume event queue. It
> is unlikely that locking is what is causing performance issues because the
> channel is heavily I/O bound. If you take a series of thread dumps, you
> will probably see that those threads are moving forward and the ones
> reading/writing from/to disk are the ones which are slower. These locks are
> unlikely to hit performance much.
>
> Thanks,
> Hari
>
> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
>
> Hi,
>
> Spent some more time debugging issues with FileChannel. The problem seems
> to lock contention reading from FlumeEventQueue:
>
> I see a lot of threads like this:
> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
> [0x00007f821afee000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at
> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>         - waiting to lock <0x0000000518ee4c90> (a
> org.apache.flume.channel.file.FlumeEventQueue)
>         at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>         at
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>         at
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>         at
> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>         at
> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:662)
>
>
> I have two file channels and 8 Avro Sinks per file channel. I added more
> sinks because they weren't draining fast enough. It seems like they send
> the batch then wait for an ack before sending again, thus sends are not
> pipelined and having more sinks seemed like a good way of getting some
> parallelism.
>
> Here's the full stack trace:
> 2013-08-13 15:30:32
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02 mixed mode):
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000056d932120> (a
> java.util.concurrent.CountDownLatch$Sync)
>  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>  at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>  at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
>  at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
>  at
> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
> at $Proxy7.appendBatch(Unknown Source)
>  at
> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
> at
> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
>  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000 nid=0x41a1 waiting
> on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>  at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005194c3340> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191a16d8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519174e08> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519174e98> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005195f9670> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519174ce8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191a1918> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191e7f60> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153 runnable
> [0x00007f8210e4d000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
> - locked <0x000000054a2917a8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519174f58> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146 runnable
> [0x00007f8210f4e000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
> - locked <0x0000000546e5dd28> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118 runnable
> [0x00007f821104f000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
> - locked <0x0000000544f763c0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111 runnable
> [0x00007f8211150000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
> - locked <0x000000054473d5a8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110 runnable
> [0x00007f8211251000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
> - locked <0x00000005437f4f70> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f runnable
> [0x00007f8211352000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
> - locked <0x00000005422a9018> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e runnable
> [0x00007f8211453000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
> - locked <0x0000000541063d50> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d runnable
> [0x00007f8211554000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
> - locked <0x0000000540dc1110> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a runnable
> [0x00007f8211655000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
> - locked <0x000000053fbbc3b0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109 runnable
> [0x00007f8211756000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
> - locked <0x000000053f812860> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd runnable
> [0x00007f8211857000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
> - locked <0x000000053e6e5570> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519175138> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519174c28> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191751c8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0 runnable
> [0x00007f8211c5b000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
> - locked <0x0000000536579db0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191750a8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1 runnable
> [0x00007f8211e5d000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
> - locked <0x00000005313943a8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051936f460> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051936dfa8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de runnable
> [0x00007f8212160000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
> - locked <0x000000052dad4468> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd runnable
> [0x00007f8212261000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
> - locked <0x000000052c73c5f0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc runnable
> [0x00007f8212362000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
> - locked <0x000000052b8da998> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db waiting for
> monitor entry [0x00007f8212462000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
> - waiting to lock <0x0000000518ee4c90> (a
> org.apache.flume.channel.file.FlumeEventQueue)
>  at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
> at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>  at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>  at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>  at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051936df18> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518fe7ae8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8 runnable
> [0x00007f8212766000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
> - locked <0x00000005263d6ba8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7 runnable
> [0x00007f8212867000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
> - locked <0x0000000525d1d0e0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6 runnable
> [0x00007f8212968000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
> - locked <0x0000000525a689d8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5 runnable
> [0x00007f8212a69000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
> - locked <0x0000000524e1e7d8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4 runnable
> [0x00007f8212b6a000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
> - locked <0x0000000524d437a8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-36" prio=10 tid=0x00007f85843ad800 nid=0x40d3 runnable
> [0x00007f8212c6b000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000524cf4140> (a sun.nio.ch.Util$2)
> - locked <0x0000000524cf4128> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000524ab32e8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-35" prio=10 tid=0x00007f85d83a9000 nid=0x40d2 runnable
> [0x00007f8212d6c000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005244877c8> (a sun.nio.ch.Util$2)
> - locked <0x00000005244877b0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000523f5b660> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-34" prio=10 tid=0x00007f85d838e000 nid=0x40d1 runnable
> [0x00007f8212e6d000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052419e490> (a sun.nio.ch.Util$2)
> - locked <0x000000052419e478> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000523f5b6f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-34" prio=10 tid=0x00007f8588390800 nid=0x40d0 waiting for
> monitor entry [0x00007f8212f6d000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
> - waiting to lock <0x0000000519111590> (a
> org.apache.flume.channel.file.LogFileV3$Writer)
>  at org.apache.flume.channel.file.Log.put(Log.java:507)
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
>  at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>  at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>  at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>  at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>  at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-35" prio=10 tid=0x00007f85843ab800 nid=0x40cf runnable
> [0x00007f821306f000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000523d40938> (a sun.nio.ch.Util$2)
> - locked <0x0000000523d40aa0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000523ecec18> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-37" prio=10 tid=0x00007f85803e3800 nid=0x40ce waiting for
> monitor entry [0x00007f821316f000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
> - waiting to lock <0x0000000518ee4c90> (a
> org.apache.flume.channel.file.FlumeEventQueue)
>  at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
> at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>  at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>  at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-34" prio=10 tid=0x00007f8584391000 nid=0x40cd runnable
> [0x00007f8213271000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000523935540> (a sun.nio.ch.Util$2)
>  - locked <0x0000000523935528> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000523788ed8> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-33" prio=10 tid=0x00007f85d8373800 nid=0x40cc runnable
> [0x00007f8213372000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005239ec658> (a sun.nio.ch.Util$2)
>  - locked <0x00000005239ec640> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000523845410> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-33" prio=10 tid=0x00007f8588375800 nid=0x40cb waiting for
> monitor entry [0x00007f8213472000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
>  - waiting to lock <0x0000000519111590> (a
> org.apache.flume.channel.file.LogFileV3$Writer)
> at org.apache.flume.channel.file.Log.put(Log.java:507)
>  at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
> at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>  at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>  at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-33" prio=10 tid=0x00007f8584376000 nid=0x40ca runnable
> [0x00007f8213574000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005237e3028> (a sun.nio.ch.Util$2)
>  - locked <0x00000005237e3010> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000523788e90> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-32" prio=10 tid=0x00007f85d8358800 nid=0x40c9 waiting for
> monitor entry [0x00007f8213674000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>  - waiting to lock <0x0000000518ee4c90> (a
> org.apache.flume.channel.file.FlumeEventQueue)
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>  at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>  at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>  at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>  at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>  at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-32" prio=10 tid=0x00007f858835b000 nid=0x40c8 runnable
> [0x00007f8213776000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000523343688> (a sun.nio.ch.Util$2)
> - locked <0x0000000523343670> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000052305e5c8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-36" prio=10 tid=0x00007f85803c9000 nid=0x40c7 runnable
> [0x00007f8213877000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005230fa580> (a sun.nio.ch.Util$2)
> - locked <0x00000005230fa568> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000522f5ca50> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-32" prio=10 tid=0x00007f858435b000 nid=0x40c6 runnable
> [0x00007f8213978000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005231367a8> (a sun.nio.ch.Util$2)
> - locked <0x0000000523136790> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000052305e658> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-31" prio=10 tid=0x00007f85d833e000 nid=0x40c5 runnable
> [0x00007f8213a79000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000522db30a0> (a sun.nio.ch.Util$2)
> - locked <0x0000000522db3088> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000522bda430> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-31" prio=10 tid=0x00007f8588340000 nid=0x40c4 runnable
> [0x00007f8213b7a000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000522c16518> (a sun.nio.ch.Util$2)
> - locked <0x0000000522c16500> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000522bd97e8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-30" prio=10 tid=0x00007f85d8323800 nid=0x40c3 runnable
> [0x00007f8213c7b000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000522bd1a20> (a sun.nio.ch.Util$2)
> - locked <0x0000000522bd1a08> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000522bda3a0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-31" prio=10 tid=0x00007f8584340800 nid=0x40c2 runnable
> [0x00007f8213d7c000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005228d1608> (a sun.nio.ch.Util$2)
> - locked <0x00000005228d15f0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005228a6aa0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-30" prio=10 tid=0x00007f8588325000 nid=0x40c1 runnable
> [0x00007f8213e7d000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052277b088> (a sun.nio.ch.Util$2)
> - locked <0x000000052277b070> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005225ed3b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-35" prio=10 tid=0x00007f85803ae000 nid=0x40c0 runnable
> [0x00007f8213f7e000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000522737380> (a sun.nio.ch.Util$2)
> - locked <0x0000000522737368> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005225ed328> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-29" prio=10 tid=0x00007f85d8308800 nid=0x40bf runnable
> [0x00007f821407f000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005226034a8> (a sun.nio.ch.Util$2)
> - locked <0x0000000522603490> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005225ed298> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-28" prio=10 tid=0x00007f85d82ed800 nid=0x40be runnable
> [0x00007f8214180000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000522359408> (a sun.nio.ch.Util$2)
> - locked <0x00000005223593f0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005221e7278> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-27" prio=10 tid=0x00007f85d82d2800 nid=0x40bd runnable
> [0x00007f8214281000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005222c0478> (a sun.nio.ch.Util$2)
> - locked <0x00000005222c0460> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005221e6e40> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-29" prio=10 tid=0x00007f858830a800 nid=0x40bc runnable
> [0x00007f8214382000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000521ddf488> (a sun.nio.ch.Util$2)
> - locked <0x0000000521ddf470> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000521c88d38> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-30" prio=10 tid=0x00007f8584326000 nid=0x40bb waiting for
> monitor entry [0x00007f8214482000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
> - waiting to lock <0x0000000519111590> (a
> org.apache.flume.channel.file.LogFileV3$Writer)
>  at org.apache.flume.channel.file.Log.put(Log.java:507)
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
>  at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>  at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>  at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>  at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>  at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-26" prio=10 tid=0x00007f85d82b8000 nid=0x40ba runnable
> [0x00007f8214584000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000521cbcb90> (a sun.nio.ch.Util$2)
> - locked <0x0000000521cbcb78> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000521279250> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-28" prio=10 tid=0x00007f85882ef800 nid=0x40b9 runnable
> [0x00007f8214685000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000521a1c2d0> (a sun.nio.ch.Util$2)
> - locked <0x0000000521a1c2b8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005218f1508> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-34" prio=10 tid=0x00007f8580393800 nid=0x40b8 runnable
> [0x00007f8214786000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052181ce58> (a sun.nio.ch.Util$2)
> - locked <0x000000052181ce40> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005212791c0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-29" prio=10 tid=0x00007f858430b800 nid=0x40b7 runnable
> [0x00007f8214887000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005219175f0> (a sun.nio.ch.Util$2)
> - locked <0x0000000521746090> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000521278e60> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-27" prio=10 tid=0x00007f85882d4800 nid=0x40b6 runnable
> [0x00007f8214988000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052163e9c8> (a sun.nio.ch.Util$2)
> - locked <0x000000052163e9b0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000521279130> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-33" prio=10 tid=0x00007f8580378800 nid=0x40b5 runnable
> [0x00007f8214a89000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052152cb38> (a sun.nio.ch.Util$2)
> - locked <0x000000052152cb20> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000521278d40> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-28" prio=10 tid=0x00007f85842f0800 nid=0x40b4 runnable
> [0x00007f8214b8a000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005213ba3e0> (a sun.nio.ch.Util$2)
> - locked <0x00000005213ba3c8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000521278020> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-27" prio=10 tid=0x00007f85842d5800 nid=0x40b3 runnable
> [0x00007f8214c8b000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005213ed660> (a sun.nio.ch.Util$2)
> - locked <0x00000005213ed648> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000521278c20> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-26" prio=10 tid=0x00007f85882b9800 nid=0x40b2 runnable
> [0x00007f8214d8c000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052133b7c0> (a sun.nio.ch.Util$2)
> - locked <0x000000052133b7a8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005212790a0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-32" prio=10 tid=0x00007f858035d800 nid=0x40af runnable
> [0x00007f8214e8d000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000520e53be0> (a sun.nio.ch.Util$2)
> - locked <0x0000000520e53bc8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000520f3bb50> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-25" prio=10 tid=0x00007f85d829d000 nid=0x40ae runnable
> [0x00007f8214f8e000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000520e09a30> (a sun.nio.ch.Util$2)
> - locked <0x0000000520e09a18> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000520f3b6d0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-31" prio=10 tid=0x00007f8580342800 nid=0x40ad runnable
> [0x00007f821508f000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000520ef70e0> (a sun.nio.ch.Util$2)
> - locked <0x0000000520ef70c8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000520f3b208> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-26" prio=10 tid=0x00007f85842ba800 nid=0x40ac runnable
> [0x00007f8215190000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000520c9dfc0> (a sun.nio.ch.Util$2)
> - locked <0x0000000520c9dfa8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000520bb3cc0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-25" prio=10 tid=0x00007f85842a0000 nid=0x40ab runnable
> [0x00007f8215291000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000520b941a8> (a sun.nio.ch.Util$2)
> - locked <0x0000000520b94190> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000520bb30c0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-24" prio=10 tid=0x00007f85d8282800 nid=0x40aa runnable
> [0x00007f8215392000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000520952288> (a sun.nio.ch.Util$2)
> - locked <0x0000000520952270> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000520356888> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-30" prio=10 tid=0x00007f8580328000 nid=0x40a9 runnable
> [0x00007f8215493000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000520a2fa20> (a sun.nio.ch.Util$2)
> - locked <0x0000000520a2fa08> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000520983da0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-23" prio=10 tid=0x00007f85d8267800 nid=0x40a8 runnable
> [0x00007f8215594000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052095c470> (a sun.nio.ch.Util$2)
> - locked <0x000000052095c458> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000520356840> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-29" prio=10 tid=0x00007f858030d000 nid=0x40a7 runnable
> [0x00007f8215695000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052065b178> (a sun.nio.ch.Util$2)
> - locked <0x000000052065b160> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000520357320> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-28" prio=10 tid=0x00007f85802f2800 nid=0x40a6 runnable
> [0x00007f8215796000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052043cad0> (a sun.nio.ch.Util$2)
> - locked <0x000000052043cab8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005203567b0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-25" prio=10 tid=0x00007f858829f000 nid=0x40a5 runnable
> [0x00007f8215897000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052041a8a8> (a sun.nio.ch.Util$2)
> - locked <0x000000052041a890> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000520356690> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-22" prio=10 tid=0x00007f85d824c800 nid=0x40a4 runnable
> [0x00007f8215998000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052036b380> (a sun.nio.ch.Util$2)
> - locked <0x000000052036b368> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000520357290> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-24" prio=10 tid=0x00007f8584285800 nid=0x40a3 runnable
> [0x00007f8215a99000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052007ac40> (a sun.nio.ch.Util$2)
> - locked <0x000000052007ac28> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051fe41948> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-27" prio=10 tid=0x00007f85802d8000 nid=0x40a2 runnable
> [0x00007f8215b9a000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005201a2ae8> (a sun.nio.ch.Util$2)
> - locked <0x00000005201a2ad0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ffda458> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-21" prio=10 tid=0x00007f85d8232000 nid=0x40a1 waiting for
> monitor entry [0x00007f8215c9a000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
> - waiting to lock <0x0000000518ee4c90> (a
> org.apache.flume.channel.file.FlumeEventQueue)
>  at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
> at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>  at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>  at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-26" prio=10 tid=0x00007f85802bd800 nid=0x40a0 runnable
> [0x00007f8215d9c000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051ff20e78> (a sun.nio.ch.Util$2)
>  - locked <0x000000051ff20e60> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051fe41900> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-24" prio=10 tid=0x00007f8588284000 nid=0x409f runnable
> [0x00007f8215e9d000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051fbc1ca0> (a sun.nio.ch.Util$2)
>  - locked <0x000000051fbc1c88> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f9a7978> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-25" prio=10 tid=0x00007f85802a2800 nid=0x409e runnable
> [0x00007f8215f9e000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051fa3b5b8> (a sun.nio.ch.Util$2)
>  - locked <0x000000051fa3b5a0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f3155b0> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-20" prio=10 tid=0x00007f85d8217800 nid=0x409d runnable
> [0x00007f821609f000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051f83c838> (a sun.nio.ch.Util$2)
>  - locked <0x000000051f83c820> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f3150e8> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-23" prio=10 tid=0x00007f8588269800 nid=0x409c runnable
> [0x00007f82161a0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051f78b9b0> (a sun.nio.ch.Util$2)
>  - locked <0x000000051f78b998> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f3156d0> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-22" prio=10 tid=0x00007f858824e800 nid=0x409b runnable
> [0x00007f82162a1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051f7c0cc8> (a sun.nio.ch.Util$2)
>  - locked <0x000000051f7c0cb0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f3151c0> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-23" prio=10 tid=0x00007f858426a800 nid=0x409a runnable
> [0x00007f82163a2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051f6065a8> (a sun.nio.ch.Util$2)
>  - locked <0x000000051f606590> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f3d9668> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-22" prio=10 tid=0x00007f858424f800 nid=0x4099 runnable
> [0x00007f82164a3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051f56d710> (a sun.nio.ch.Util$2)
>  - locked <0x000000051f56d6f8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f314f80> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-21" prio=10 tid=0x00007f8588233800 nid=0x4098 runnable
> [0x00007f82165a4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051f3bcd68> (a sun.nio.ch.Util$2)
>  - locked <0x000000051f3bcd50> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f315400> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-19" prio=10 tid=0x00007f85d81fc800 nid=0x4097 runnable
> [0x00007f82166a5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051f53b198> (a sun.nio.ch.Util$2)
>  - locked <0x000000051f53b180> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f315760> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-20" prio=10 tid=0x00007f8588218800 nid=0x4096 waiting for
> monitor entry [0x00007f82167a5000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
>  - waiting to lock <0x0000000519111590> (a
> org.apache.flume.channel.file.LogFileV3$Writer)
> at org.apache.flume.channel.file.Log.put(Log.java:507)
>  at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
> at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>  at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>  at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-18" prio=10 tid=0x00007f85d81e1800 nid=0x4095 runnable
> [0x00007f82168a7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051ee92ba0> (a sun.nio.ch.Util$2)
>  - locked <0x000000051ee92b88> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051ed19740> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-19" prio=10 tid=0x00007f85881fd800 nid=0x4094 runnable
> [0x00007f82169a8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051ee5da40> (a sun.nio.ch.Util$2)
>  - locked <0x000000051ee5da28> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051e77a940> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-17" prio=10 tid=0x00007f85d81c6800 nid=0x4093 runnable
> [0x00007f8216aa9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051ecb1730> (a sun.nio.ch.Util$2)
>  - locked <0x000000051ecb1718> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051e77a8b0> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-21" prio=10 tid=0x00007f8584234800 nid=0x4092 runnable
> [0x00007f8216baa000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051e899660> (a sun.nio.ch.Util$2)
>  - locked <0x000000051e899648> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051e77a310> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-24" prio=10 tid=0x00007f8580287000 nid=0x4091 runnable
> [0x00007f8216caa000]
>    java.lang.Thread.State: RUNNABLE
> at
> org.apache.flume.channel.file.FlumeEventQueue.addTail(FlumeEventQueue.java:165)
>  - locked <0x0000000518ee4c90> (a
> org.apache.flume.channel.file.FlumeEventQueue)
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:529)
>  - locked <0x0000000518ee4c90> (a
> org.apache.flume.channel.file.FlumeEventQueue)
> at
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
>  at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>  at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>  at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>  at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-23" prio=10 tid=0x00007f858026c000 nid=0x4090 runnable
> [0x00007f8216dac000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051ea8b6f0> (a sun.nio.ch.Util$2)
> - locked <0x000000051ea8b6d8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051e77a820> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-16" prio=10 tid=0x00007f85d81ac000 nid=0x408f runnable
> [0x00007f8216ead000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051e7c8808> (a sun.nio.ch.Util$2)
> - locked <0x000000051e7c87f0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051e77a280> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-20" prio=10 tid=0x00007f8584219800 nid=0x408e runnable
> [0x00007f8216fae000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051e8fffa8> (a sun.nio.ch.Util$2)
> - locked <0x000000051e8fff90> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051e77a700> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-22" prio=10 tid=0x00007f8580251000 nid=0x408d runnable
> [0x00007f82170af000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051e4dc398> (a sun.nio.ch.Util$2)
> - locked <0x000000051e4dc380> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051dcbeef8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-19" prio=10 tid=0x00007f85841fe800 nid=0x408c runnable
> [0x00007f82171b0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051e221520> (a sun.nio.ch.Util$2)
> - locked <0x000000051e221508> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051e407860> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-18" prio=10 tid=0x00007f85841e3800 nid=0x408b runnable
> [0x00007f82172b1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051e465298> (a sun.nio.ch.Util$2)
> - locked <0x000000051e465280> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051dcbef88> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-18" prio=10 tid=0x00007f85881e3000 nid=0x408a runnable
> [0x00007f82173b2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051dea12e0> (a sun.nio.ch.Util$2)
> - locked <0x000000051dea12c8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051cfd4d78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-15" prio=10 tid=0x00007f85d8191000 nid=0x4089 runnable
> [0x00007f82174b3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051de322b8> (a sun.nio.ch.Util$2)
> - locked <0x000000051de322a0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051dcbec28> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-14" prio=10 tid=0x00007f85d8176800 nid=0x4088 runnable
> [0x00007f82175b4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051dea29a8> (a sun.nio.ch.Util$2)
> - locked <0x000000051dea2990> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051dcbf018> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-21" prio=10 tid=0x00007f8580236800 nid=0x4087 runnable
> [0x00007f82176b5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051dca95d0> (a sun.nio.ch.Util$2)
> - locked <0x000000051dca95b8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051cfd4e08> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-17" prio=10 tid=0x00007f85841c8800 nid=0x4086 runnable
> [0x00007f82177b6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051da8d058> (a sun.nio.ch.Util$2)
> - locked <0x000000051da8d040> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051cd693f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-20" prio=10 tid=0x00007f858021b800 nid=0x4085 runnable
> [0x00007f82178b7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051d95a5b0> (a sun.nio.ch.Util$2)
> - locked <0x000000051d95a598> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051cfd4e98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-16" prio=10 tid=0x00007f85841ad800 nid=0x4083 runnable
> [0x00007f82179b8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051d92a278> (a sun.nio.ch.Util$2)
> - locked <0x000000051d92a260> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051cfd4fb8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-13" prio=10 tid=0x00007f85d815b800 nid=0x4082 runnable
> [0x00007f8217ab9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051d734788> (a sun.nio.ch.Util$2)
> - locked <0x000000051d734770> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051cd69488> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-19" prio=10 tid=0x00007f8580200800 nid=0x4081 runnable
> [0x00007f8217bba000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051d4268e8> (a sun.nio.ch.Util$2)
> - locked <0x000000051d4268d0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051d139810> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-15" prio=10 tid=0x00007f8584192800 nid=0x4080 runnable
> [0x00007f8217cbb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051d4cda88> (a sun.nio.ch.Util$2)
> - locked <0x000000051d4cda70> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051babcf98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-17" prio=10 tid=0x00007f85881c8000 nid=0x407f runnable
> [0x00007f8217dbc000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051d32ffe8> (a sun.nio.ch.Util$2)
> - locked <0x000000051d32ffd0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051cd691b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-14" prio=10 tid=0x00007f8584177800 nid=0x407e runnable
> [0x00007f8217ebd000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051d506c60> (a sun.nio.ch.Util$2)
> - locked <0x000000051d506c48> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051cfd5048> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-16" prio=10 tid=0x00007f85881ad000 nid=0x407d runnable
> [0x00007f8217fbe000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051d1223e0> (a sun.nio.ch.Util$2)
> - locked <0x000000051d1223c8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051c6c7b40> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-15" prio=10 tid=0x00007f8588192800 nid=0x407c runnable
> [0x00007f82180bf000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051d146f50> (a sun.nio.ch.Util$2)
> - locked <0x000000051d146f38> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051cd69248> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-13" prio=10 tid=0x00007f858415c800 nid=0x407b runnable
> [0x00007f82181c0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051d11d1b8> (a sun.nio.ch.Util$2)
> - locked <0x000000051d11d1a0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051c6c7c18> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-12" prio=10 tid=0x00007f85d8141000 nid=0x407a runnable
> [0x00007f82182c1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051cd01a78> (a sun.nio.ch.Util$2)
> - locked <0x000000051cd01a60> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051cd695a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-11" prio=10 tid=0x00007f85d8126800 nid=0x4079 runnable
> [0x00007f82183c2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051cfa6f88> (a sun.nio.ch.Util$2)
> - locked <0x000000051cfa6f70> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ba67370> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-14" prio=10 tid=0x00007f8588177800 nid=0x4078 runnable
> [0x00007f82184c3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051ce5cdd0> (a sun.nio.ch.Util$2)
> - locked <0x000000051ce5cdb8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051c6c7bd0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-12" prio=10 tid=0x00007f8584142000 nid=0x4077 runnable
> [0x00007f82185c4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051cc27cb8> (a sun.nio.ch.Util$2)
> - locked <0x000000051cc27ca0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051babd0b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-18" prio=10 tid=0x00007f85801e6000 nid=0x4076 runnable
> [0x00007f82186c5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051cb895c8> (a sun.nio.ch.Util$2)
> - locked <0x000000051cc77fd0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051c6c7cf0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-11" prio=10 tid=0x00007f8584127800 nid=0x4075 runnable
> [0x00007f82187c6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051ca540d0> (a sun.nio.ch.Util$2)
> - locked <0x000000051caad368> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051c6c7e10> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-13" prio=10 tid=0x00007f858815c800 nid=0x4074 runnable
> [0x00007f82188c7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051c953ab8> (a sun.nio.ch.Util$2)
> - locked <0x000000051ca799a8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051babd028> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-17" prio=10 tid=0x00007f85801cb800 nid=0x4073 runnable
> [0x00007f82189c8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051c77c3d8> (a sun.nio.ch.Util$2)
> - locked <0x000000051c8137c8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051c3856f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-10" prio=10 tid=0x00007f858410c800 nid=0x4072 runnable
> [0x00007f8218ac9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051c7dd2b0> (a sun.nio.ch.Util$2)
> - locked <0x000000051c613988> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051babcde8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-16" prio=10 tid=0x00007f85801b1000 nid=0x4071 runnable
> [0x00007f8218bca000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051c776580> (a sun.nio.ch.Util$2)
> - locked <0x000000051c6633c8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051c6c7ea0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-9" prio=10 tid=0x00007f85840f1800 nid=0x4070 runnable
> [0x00007f8218ccb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051c36b550> (a sun.nio.ch.Util$2)
> - locked <0x000000051c36b568> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ba675f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-8" prio=10 tid=0x00007f85840d6800 nid=0x406f runnable
> [0x00007f8218dcc000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051c546b00> (a sun.nio.ch.Util$2)
> - locked <0x000000051c40a178> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ba67760> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-7" prio=10 tid=0x00007f85840bb800 nid=0x406e runnable
> [0x00007f8218ecd000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051c2ecf90> (a sun.nio.ch.Util$2)
> - locked <0x000000051c29a100> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ba675b0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-15" prio=10 tid=0x00007f8580196000 nid=0x406d runnable
> [0x00007f8218fce000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051c2ee670> (a sun.nio.ch.Util$2)
> - locked <0x000000051c2ee688> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ba67718> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-12" prio=10 tid=0x00007f8588141800 nid=0x406c runnable
> [0x00007f82190cf000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051c0bd978> (a sun.nio.ch.Util$2)
> - locked <0x000000051c16cc00> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051babcf08> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-6" prio=10 tid=0x00007f85840a0800 nid=0x406b runnable
> [0x00007f82191d0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051c068508> (a sun.nio.ch.Util$2)
> - locked <0x000000051bf825f0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ba676d0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-11" prio=10 tid=0x00007f8588127000 nid=0x406a runnable
> [0x00007f82192d1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051becaa68> (a sun.nio.ch.Util$2)
> - locked <0x000000051c03ead0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051babce78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-10" prio=10 tid=0x00007f858810c800 nid=0x4069 runnable
> [0x00007f82193d2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051be31b98> (a sun.nio.ch.Util$2)
> - locked <0x000000051bd56790> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ba67490> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-14" prio=10 tid=0x00007f858017b000 nid=0x4068 runnable
> [0x00007f82194d3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051bccb8c0> (a sun.nio.ch.Util$2)
> - locked <0x000000051bb14420> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051babcd58> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-5" prio=10 tid=0x00007f8584085800 nid=0x4067 runnable
> [0x00007f82195d4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051bd004a8> (a sun.nio.ch.Util$2)
> - locked <0x000000051bd49480> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ba67400> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-4" prio=10 tid=0x00007f858406b000 nid=0x4066 runnable
> [0x00007f82196d5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051ba1e928> (a sun.nio.ch.Util$2)
> - locked <0x000000051bccdd00> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ba66ef0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-9" prio=10 tid=0x00007f85880f1800 nid=0x4065 waiting for
> monitor entry [0x00007f82197d5000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
> - waiting to lock <0x0000000519111590> (a
> org.apache.flume.channel.file.LogFileV3$Writer)
>  at org.apache.flume.channel.file.Log.put(Log.java:507)
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
>  at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>  at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>  at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>  at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>  at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-8" prio=10 tid=0x00007f85880d6800 nid=0x4064 runnable
> [0x00007f82198d7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051b5bcca8> (a sun.nio.ch.Util$2)
> - locked <0x000000051b7ef8f0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051b5c7cb8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-7" prio=10 tid=0x00007f85880bb800 nid=0x4063 runnable
> [0x00007f82199d8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051b788ce8> (a sun.nio.ch.Util$2)
> - locked <0x000000051b788d00> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051b285260> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-13" prio=10 tid=0x00007f8580160800 nid=0x4062 runnable
> [0x00007f8219ad9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051b71a0f8> (a sun.nio.ch.Util$2)
> - locked <0x000000051b750e98> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051aed5a70> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-10" prio=10 tid=0x00007f85d810b800 nid=0x4061 runnable
> [0x00007f8219bda000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051b517878> (a sun.nio.ch.Util$2)
> - locked <0x000000051b5bab30> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051aed5b90> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-9" prio=10 tid=0x00007f85d80f0800 nid=0x4060 runnable
> [0x00007f8219cdb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051b5656e0> (a sun.nio.ch.Util$2)
> - locked <0x000000051b5657a0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051b285338> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-8" prio=10 tid=0x00007f85d80d5800 nid=0x405f runnable
> [0x00007f8219ddc000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051b3df220> (a sun.nio.ch.Util$2)
> - locked <0x000000051b475118> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051aed5c20> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-6" prio=10 tid=0x00007f85880a0800 nid=0x405e runnable
> [0x00007f8219edd000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051b11c010> (a sun.nio.ch.Util$2)
> - locked <0x000000051b1eab78> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051af91478> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-7" prio=10 tid=0x00007f85d80bb000 nid=0x405d runnable
> [0x00007f8219fde000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051b1d36e0> (a sun.nio.ch.Util$2)
> - locked <0x000000051b0328c0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051aed5cb0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-12" prio=10 tid=0x00007f8580145800 nid=0x405c runnable
> [0x00007f821a0df000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051aeb3070> (a sun.nio.ch.Util$2)
> - locked <0x000000051aeb3088> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a98a820> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-6" prio=10 tid=0x00007f85d80a0000 nid=0x405b runnable
> [0x00007f821a1e0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051aefa920> (a sun.nio.ch.Util$2)
> - locked <0x000000051af3ee40> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ac28810> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-5" prio=10 tid=0x00007f8588086000 nid=0x405a runnable
> [0x00007f821a2e1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051aefe1b0> (a sun.nio.ch.Util$2)
> - locked <0x000000051ad2b858> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051aed5e18> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-11" prio=10 tid=0x00007f858012a800 nid=0x4059 runnable
> [0x00007f821a3e2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051ac42588> (a sun.nio.ch.Util$2)
> - locked <0x000000051ad05d70> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a98a7d8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-3" prio=10 tid=0x00007f8584050000 nid=0x4058 runnable
> [0x00007f821a4e3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051ad06100> (a sun.nio.ch.Util$2)
> - locked <0x000000051ad47428> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ac289c0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-10" prio=10 tid=0x00007f858010f800 nid=0x4057 runnable
> [0x00007f821a5e4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051aab5d80> (a sun.nio.ch.Util$2)
> - locked <0x000000051ab8ce38> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a98a748> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-5" prio=10 tid=0x00007f85d8085800 nid=0x4056 runnable
> [0x00007f821a6e5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a962980> (a sun.nio.ch.Util$2)
> - locked <0x000000051a775fb0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a842098> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-2" prio=10 tid=0x00007f8584035000 nid=0x4055 runnable
> [0x00007f821a7e6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a817210> (a sun.nio.ch.Util$2)
> - locked <0x000000051a7d2b48> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a841ca8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-4" prio=10 tid=0x00007f85d806b000 nid=0x4054 runnable
> [0x00007f821a8e7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a903340> (a sun.nio.ch.Util$2)
> - locked <0x000000051a903358> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a42db50> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-9" prio=10 tid=0x00007f85800f4800 nid=0x4053 runnable
> [0x00007f821a9e8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a4f17d8> (a sun.nio.ch.Util$2)
> - locked <0x000000051a37ae08> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a22e340> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-4" prio=10 tid=0x00007f858806b000 nid=0x4052 runnable
> [0x00007f821aae9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a4d81e8> (a sun.nio.ch.Util$2)
> - locked <0x000000051a616bb0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a1dfb40> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-3" prio=10 tid=0x00007f85d8050000 nid=0x4051 runnable
> [0x00007f821abea000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a4fb240> (a sun.nio.ch.Util$2)
> - locked <0x000000051a3913b8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a213878> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-2" prio=10 tid=0x00007f85d8035800 nid=0x4050 runnable
> [0x00007f821aceb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a506248> (a sun.nio.ch.Util$2)
> - locked <0x000000051a506308> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a42dd48> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-8" prio=10 tid=0x00007f85800d9800 nid=0x404f runnable
> [0x00007f821adec000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a2d5170> (a sun.nio.ch.Util$2)
> - locked <0x000000051a2d5188> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a22e2b0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-7" prio=10 tid=0x00007f85800be800 nid=0x404e runnable
> [0x00007f821aeed000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a2d3cb8> (a sun.nio.ch.Util$2)
> - locked <0x000000051a2b8310> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a213758> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
> [0x00007f821afee000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
> - waiting to lock <0x0000000518ee4c90> (a
> org.apache.flume.channel.file.FlumeEventQueue)
>  at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
> at
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>  at
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>  at
> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f836c01a000
> nid=0x404c runnable [0x00007f821b0ef000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a045bb0> (a sun.nio.ch.Util$2)
> - locked <0x000000051a045bc8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a000c98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b376800
> nid=0x404b waiting on condition [0x00007f821b1f0000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051a045c10> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.NativeThread.current(Native Method)
> at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>  at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
> - locked <0x00000005190ec998> (a java.lang.Object)
>  at org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
> at org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>  - locked <0x0000000519111590> (a
> org.apache.flume.channel.file.LogFileV3$Writer)
> at org.apache.flume.channel.file.Log.take(Log.java:550)
>  at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
> at
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>  at
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>  at
> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8354035000
> nid=0x4049 runnable [0x00007f821b3f2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a045c28> (a sun.nio.ch.Util$2)
> - locked <0x000000051a045c40> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a000ce0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834801a000
> nid=0x4048 runnable [0x00007f821b4f3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a17bd60> (a sun.nio.ch.Util$2)
> - locked <0x000000051a17bd78> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a180230> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b374800
> nid=0x4047 waiting on condition [0x00007f821b5f4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051a08ba88> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857338a800
> nid=0x4046 waiting on condition [0x00007f821b6f5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051a147940> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
> tid=0x00007f85f3bff800 nid=0x4045 waiting on condition [0x00007f821b7f6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000056dca65c0> (a
> java.util.concurrent.FutureTask$Sync)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
>  at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:227)
>  at java.util.concurrent.FutureTask.get(FutureTask.java:91)
> at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:323)
>  at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:269)
> at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:352)
>  at
> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8344035000
> nid=0x4044 runnable [0x00007f821b8f7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a184a90> (a sun.nio.ch.Util$2)
> - locked <0x000000051a184aa8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a0bb2e0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bfd800
> nid=0x4043 waiting on condition [0x00007f821b9f8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051a147958> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f833001a000
> nid=0x4042 runnable [0x00007f821baf9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a188f28> (a sun.nio.ch.Util$2)
> - locked <0x000000051a188f40> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519d90758> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8334035000
> nid=0x4041 runnable [0x00007f821bbfa000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a045cd0> (a sun.nio.ch.Util$2)
> - locked <0x000000051a045ce8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a000d28> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b372800
> nid=0x4040 waiting on condition [0x00007f821bcfb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f74590> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573388800
> nid=0x403f waiting on condition [0x00007f821bdfc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fb0c90> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832c035000
> nid=0x403e runnable [0x00007f821befd000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a045d30> (a sun.nio.ch.Util$2)
> - locked <0x000000051a045d48> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a000d70> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bfb800
> nid=0x403d waiting on condition [0x00007f821bffe000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519e5d388> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f831801a000
> nid=0x403c runnable [0x00007f82341c0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519ebdb70> (a sun.nio.ch.Util$2)
> - locked <0x0000000519ebdb88> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519d3a778> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f831001a000
> nid=0x403b runnable [0x00007f82342c1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519f47f90> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f47fa8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519e11ca8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaf0000
> nid=0x403a waiting on condition [0x00007f82343c2000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f73018> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8308035000
> nid=0x4039 runnable [0x00007f82344c3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519f745a8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f745c0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519ca9f78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573386800
> nid=0x4038 waiting on condition [0x00007f82345c4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fa5ce0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bf9800
> nid=0x4037 waiting on condition [0x00007f82346c5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fb0ca8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8300035000
> nid=0x4036 runnable [0x00007f82347c6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519f22d20> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f22d38> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519d9f4b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaee000
> nid=0x4035 waiting on condition [0x00007f82348c7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fb0cc0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82fc01a000
> nid=0x4034 runnable [0x00007f82349c8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519f74608> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f74620> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519ca9fc0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82f001a000
> nid=0x4033 runnable [0x00007f8234ac9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519fb0cd8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fb0cf0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519cadd40> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f44800
> nid=0x4032 waiting on condition [0x00007f8234bca000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519ebdbd0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3377000
> nid=0x4031 waiting on condition [0x00007f8234ccb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f701c0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82e8035000
> nid=0x4030 runnable [0x00007f8234dcc000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519f47ff0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f48008> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519e11cf0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaec000
> nid=0x402f waiting on condition [0x00007f8234ecd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fa5cf8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
> tid=0x00007f857fc38000 nid=0x402e waiting for monitor entry
> [0x00007f8234fce000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
> - waiting to lock <0x0000000518ee4c90> (a
> org.apache.flume.channel.file.FlumeEventQueue)
>  at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
> at
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>  at
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>  at
> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82dc01a000
> nid=0x402d runnable [0x00007f82350cf000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519ebdbe8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519ebdc00> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519d3a7c0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82d001a000
> nid=0x402c runnable [0x00007f82351d0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519fb0d38> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fb0d50> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519cadd88> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-1" prio=10 tid=0x00007f85d8001000 nid=0x402b runnable
> [0x00007f82352d1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519f481a0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f49040> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519e11d80> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6d3800
> nid=0x402a waiting on condition [0x00007f82353d2000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f22d80> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f42800
> nid=0x4029 waiting on condition [0x00007f82354d3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fa5d10> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82cc035000
> nid=0x4028 runnable [0x00007f82355d4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519fb0d98> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fb0db0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519caddd0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-3" prio=10 tid=0x00007f8588001000 nid=0x4027 runnable
> [0x00007f82356d5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519ebdc48> (a sun.nio.ch.Util$2)
> - locked <0x0000000519ebdc60> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519d3a808> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3375000
> nid=0x4026 waiting on condition [0x00007f82357d6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fa5d28> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82c4035000
> nid=0x4025 runnable [0x00007f82358d7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519f22d98> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f22db0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519d9f500> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaea000
> nid=0x4024 waiting on condition [0x00007f82359d8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fa5d40> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a8035000
> nid=0x4023 runnable [0x00007f8235ad9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519fb0df8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fb0e10> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519cade18> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82b401a000
> nid=0x4022 runnable [0x00007f8235bda000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519f73030> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f73048> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519e69de8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82ac035000
> nid=0x4021 runnable [0x00007f8235cdb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519f61380> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f61398> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005197600f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6d1800
> nid=0x4020 waiting on condition [0x00007f8235ddc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f22df8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f40800
> nid=0x401f waiting on condition [0x00007f8235edd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fa5d58> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3373000
> nid=0x401e waiting on condition [0x00007f8235fde000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519ebdca8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a4035000
> nid=0x401d runnable [0x00007f82360df000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519fb0e58> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fb0e70> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519cade60> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a8000
> nid=0x401c waiting on condition [0x00007f82361e0000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f22e10> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-6" prio=10 tid=0x00007f85800a3800 nid=0x401b runnable
> [0x00007f82362e1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519ebde28> (a sun.nio.ch.Util$2)
> - locked <0x0000000519de7350> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519d3a898> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828801a000
> nid=0x401a runnable [0x00007f82363e2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519fa5d70> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fa5d88> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519d36338> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828c035000
> nid=0x4019 runnable [0x00007f82364e3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519f73090> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f730a8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519e69e30> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8280035000
> nid=0x4018 runnable [0x00007f82365e4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519fb0eb8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519e1a7f8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519cadea8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6cf800
> nid=0x4017 waiting on condition [0x00007f82366e5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f22e28> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f3e800
> nid=0x4016 waiting on condition [0x00007f82367e6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f701d8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3371000
> nid=0x4015 waiting on condition [0x00007f82368e7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fb0f00> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8278035000
> nid=0x4014 runnable [0x00007f82369e8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519e93628> (a sun.nio.ch.Util$2)
> - locked <0x0000000519e93640> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519d9f548> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a6000
> nid=0x4013 waiting on condition [0x00007f8236ae9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005199bdaf0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f827001a000
> nid=0x4012 runnable [0x00007f8236bea000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519fb0f18> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fb0f30> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519cadef0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6cd800
> nid=0x4011 waiting on condition [0x00007f8236ceb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c47f38> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826c035000
> nid=0x4010 runnable [0x00007f8236dec000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519fa5dd0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fa5de8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519d36380> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826001a000
> nid=0x400f runnable [0x00007f8236eed000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c8c170> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c8c188> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519a73d78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572aeb000
> nid=0x400e waiting on condition [0x00007f8236fee000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c7e570> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2e800
> nid=0x400d waiting on condition [0x00007f82370ef000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c47f50> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8258035000
> nid=0x400b runnable [0x00007f82371f0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c4b190> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c4b1a8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005199eb068> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a4000
> nid=0x400a waiting on condition [0x00007f82372f1000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c47f68> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f825001a000
> nid=0x4009 runnable [0x00007f82373f2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c33550> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c33568> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051996b7b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e29c000
> nid=0x4008 waiting on condition [0x00007f82374f3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c47f80> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f824c03f800
> nid=0x4007 runnable [0x00007f82375f4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c7e588> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e5a0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005199a1910> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8240035000
> nid=0x4006 runnable [0x00007f82376f5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c47f98> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c47fb0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005199536b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae9000
> nid=0x4005 waiting on condition [0x00007f82377f6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c368c8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2c800
> nid=0x4004 waiting on condition [0x00007f82378f7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c29bf0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8238035000
> nid=0x4003 runnable [0x00007f82379f8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519bd8198> (a sun.nio.ch.Util$2)
> - locked <0x0000000519bd81b0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519b59da0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a2000
> nid=0x4002 waiting on condition [0x00007f8237af9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c7e5e8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8230035000
> nid=0x4001 runnable [0x00007f8237bfa000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c47ff8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c48010> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519953700> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e29a000
> nid=0x4000 waiting on condition [0x00007f8237cfb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c7e600> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f823001a000
> nid=0x3fff runnable [0x00007f8237dfc000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c7e618> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e630> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005199a1958> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f823c01a000
> nid=0x3ffe runnable [0x00007f8237efd000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c48058> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c48070> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519a6d798> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f823801a000
> nid=0x3ffd runnable [0x00007f8237ffe000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c335b0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c335c8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051996b800> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae7000
> nid=0x3ffc waiting on condition [0x00007f84001c0000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c480b8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2a800
> nid=0x3ffb waiting on condition [0x00007f84002c1000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519bd81f8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24e800
> nid=0x3ffa waiting on condition [0x00007f84003c2000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c29c08> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f824001a000
> nid=0x3ff9 runnable [0x00007f84004c3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c7e678> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e690> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005199a19a0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e298000
> nid=0x3ff8 waiting on condition [0x00007f84005c4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c67408> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-5" prio=10 tid=0x00007f8580088800 nid=0x3ff7 runnable
> [0x00007f84006c5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c48238> (a sun.nio.ch.Util$2)
> - locked <0x0000000519d5e0d0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519a6d828> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f825c01a000
> nid=0x3ff6 runnable [0x00007f84007c6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c368e0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c368f8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519953088> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f825801a000
> nid=0x3ff5 runnable [0x00007f84008c7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c33610> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c33628> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051996b848> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826401a000
> nid=0x3ff4 runnable [0x00007f84009c8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c7e6d8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e6f0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005199a19e8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae5000
> nid=0x3ff3 waiting on condition [0x00007f8400ac9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c730b8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24c800
> nid=0x3ff2 waiting on condition [0x00007f8400bca000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c29c20> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f29000
> nid=0x3ff1 waiting on condition [0x00007f8400ccb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c4b1f0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826c01a000
> nid=0x3ff0 runnable [0x00007f8400dcc000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519bd8210> (a sun.nio.ch.Util$2)
> - locked <0x0000000519bd8228> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519b59de8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e296800
> nid=0x3fef waiting on condition [0x00007f8400ecd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c8c1d0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f827801a000
> nid=0x3fee runnable [0x00007f8400fce000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c7e738> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e750> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005199a1a30> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828401a000
> nid=0x3fed runnable [0x00007f84010cf000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c33670> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c33688> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051996b890> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828001a000
> nid=0x3fec runnable [0x00007f84011d0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c29c38> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c29c50> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519955c68> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828c01a000
> nid=0x3feb runnable [0x00007f84012d1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c7e798> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e7b0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005199533a0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae6000
> nid=0x3fea waiting on condition [0x00007f84013d2000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519bd8270> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24a800
> nid=0x3fe9 waiting on condition [0x00007f84014d3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c36940> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572273800
> nid=0x3fe8 waiting on condition [0x00007f84015d4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c8c1e8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de54800
> nid=0x3fe7 waiting on condition [0x00007f84016d5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c67420> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f829001a000
> nid=0x3fe6 runnable [0x00007f84017d6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c7e7f8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e810> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005199533e8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a401a000
> nid=0x3fe5 runnable [0x00007f84018d7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c336d0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c336e8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051996b8d8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae4000
> nid=0x3fe4 waiting on condition [0x00007f84019d8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519946350> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-4" prio=10 tid=0x00007f858006d800 nid=0x3fe3 runnable
> [0x00007f8401ad9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519bd83d8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f042a0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519b59e78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82ac01a000
> nid=0x3fe2 runnable [0x00007f8401bda000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c7e858> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e870> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519953430> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a248800
> nid=0x3fe1 waiting on condition [0x00007f8401cdb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519890ec8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a801a000
> nid=0x3fe0 runnable [0x00007f8401ddc000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c8c200> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c8c218> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519a73dc0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572271800
> nid=0x3fdf waiting on condition [0x00007f8401edd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198a3170> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de52800
> nid=0x3fde waiting on condition [0x00007f8401fde000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198b7610> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82b801a000
> nid=0x3fdd runnable [0x00007f84020df000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519947f70> (a sun.nio.ch.Util$2)
> - locked <0x0000000519947f88> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005197bc938> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82c401a000
> nid=0x3fdc runnable [0x00007f84021e0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519946368> (a sun.nio.ch.Util$2)
> - locked <0x0000000519946380> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051983b9b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82cc01a000
> nid=0x3fdb runnable [0x00007f84022e1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005198a3188> (a sun.nio.ch.Util$2)
> - locked <0x00000005198a31a0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005197300e0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae2000
> nid=0x3fda waiting on condition [0x00007f84023e2000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005199463c8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857226f800
> nid=0x3fd9 waiting on condition [0x00007f84024e3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519792380> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82c801a000
> nid=0x3fd8 runnable [0x00007f84025e4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519947fd0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519947fe8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005197bc980> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579597000
> nid=0x3fd7 waiting on condition [0x00007f84026e5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519488cc0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de50800
> nid=0x3fd6 waiting on condition [0x00007f84027e6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198cc640> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82d801a000
> nid=0x3fd5 runnable [0x00007f84028e7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005199463e0> (a sun.nio.ch.Util$2)
> - locked <0x00000005199463f8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051983ba00> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82e401a000
> nid=0x3fd4 runnable [0x00007f84029e8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005198f7fc8> (a sun.nio.ch.Util$2)
> - locked <0x00000005198f7fe0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194d0160> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82ec01a000
> nid=0x3fd3 runnable [0x00007f8402ae9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005198a31e8> (a sun.nio.ch.Util$2)
> - locked <0x00000005198a3200> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519730128> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae0800
> nid=0x3fd2 waiting on condition [0x00007f8402bea000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519948030> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857226d800
> nid=0x3fd1 waiting on condition [0x00007f8402ceb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519946440> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82e801a000
> nid=0x3fd0 runnable [0x00007f8402dec000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519890ee0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519890ef8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005196f56f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579595000
> nid=0x3fcf waiting on condition [0x00007f8402eed000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198b7628> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de4e800
> nid=0x3fce waiting on condition [0x00007f8402fee000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198a3248> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830401a000
> nid=0x3fcd runnable [0x00007f84030ef000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519946458> (a sun.nio.ch.Util$2)
> - locked <0x0000000519946470> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051983ba48> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830001a000
> nid=0x3fcc runnable [0x00007f84031f0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519937b88> (a sun.nio.ch.Util$2)
> - locked <0x0000000519937ba0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519887cb8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830c01a000
> nid=0x3fcb runnable [0x00007f84032f1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519948048> (a sun.nio.ch.Util$2)
> - locked <0x0000000519948060> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005197bc9c8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830801a000
> nid=0x3fca runnable [0x00007f84033f2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005199464b8> (a sun.nio.ch.Util$2)
> - locked <0x00000005199464d0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051983ba90> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e2b000
> nid=0x3fc9 waiting on condition [0x00007f84034f3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519943980> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1595000
> nid=0x3fc8 waiting on condition [0x00007f84035f4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198a3260> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857bff5800
> nid=0x3fc7 waiting on condition [0x00007f84036f5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519901138> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da0c000
> nid=0x3fc6 waiting on condition [0x00007f84037f6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005199480a8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832001a000
> nid=0x3fc5 runnable [0x00007f84038f7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519946518> (a sun.nio.ch.Util$2)
> - locked <0x0000000519946530> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051983bad8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832c01a000
> nid=0x3fc4 runnable [0x00007f84039f8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005198a3278> (a sun.nio.ch.Util$2)
> - locked <0x00000005198a3290> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519730170> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832801a000
> nid=0x3fc3 runnable [0x00007f8403af9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005198f8028> (a sun.nio.ch.Util$2)
> - locked <0x00000005198f8040> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194d01a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f833401a000
> nid=0x3fc2 runnable [0x00007f8403bfa000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005198b7640> (a sun.nio.ch.Util$2)
> - locked <0x00000005198b7658> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005196f3480> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e29000
> nid=0x3fc1 waiting on condition [0x00007f8403cfb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519946578> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1593000
> nid=0x3fc0 waiting on condition [0x00007f8403dfc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005199480c0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579592000
> nid=0x3fbf waiting on condition [0x00007f8403efd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519943998> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da0a800
> nid=0x3fbe waiting on condition [0x00007f84401c0000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519946590> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834401a000
> nid=0x3fbd runnable [0x00007f84402c1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519903410> (a sun.nio.ch.Util$2)
> - locked <0x0000000519903428> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005191a1498> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834001a000
> nid=0x3fbc runnable [0x00007f84403c2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519943ef0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519943f08> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005196f5740> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834c01a000
> nid=0x3fbb runnable [0x00007f84404c3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005199497b0> (a sun.nio.ch.Util$2)
> - locked <0x00000005199497c8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519685f60> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573ff1800
> nid=0x3fba waiting on condition [0x00007f84405c4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198f8088> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1591800
> nid=0x3fb9 waiting on condition [0x00007f84406c5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198a32d8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579591000
> nid=0x3fb8 waiting on condition [0x00007f84407c6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005199480d8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f835401a000
> nid=0x3fb7 runnable [0x00007f84408c7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005199465a8> (a sun.nio.ch.Util$2)
> - locked <0x00000005199465c0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051983bb20> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da08800
> nid=0x3fb6 waiting on condition [0x00007f84409c8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005199439b0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-1" prio=10 tid=0x00007f8584001000 nid=0x3fb5 runnable
> [0x00007f8440ac9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005198b77f0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519a00960> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519952e48> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f837401a000
> nid=0x3fb4 runnable [0x00007f8440bca000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519937be8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519937c00> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519887d00> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f836801a000
> nid=0x3fb3 runnable [0x00007f8440ccb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519943f50> (a sun.nio.ch.Util$2)
> - locked <0x0000000519943f68> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005196f5788> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f837001a000
> nid=0x3fb2 runnable [0x00007f8440dcc000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519901150> (a sun.nio.ch.Util$2)
> - locked <0x0000000519901168> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519106100> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f158f800
> nid=0x3fb1 waiting on condition [0x00007f8440ecd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519946608> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e26800
> nid=0x3fb0 waiting on condition [0x00007f8440fce000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198a32f0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913e000
> nid=0x3faf waiting on condition [0x00007f84410cf000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005199439c8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f837801a000
> nid=0x3fae runnable [0x00007f84411d0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005199480f0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519948108> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005197bca10> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da07000
> nid=0x3fad waiting on condition [0x00007f84412d1000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519943fb0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f838801a000
> nid=0x3fac runnable [0x00007f84413d2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005198f80a0> (a sun.nio.ch.Util$2)
> - locked <0x00000005198f80b8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194d01f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f839401a000
> nid=0x3fab runnable [0x00007f84414d3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519946620> (a sun.nio.ch.Util$2)
> - locked <0x0000000519946638> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051983bb68> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f839001a000
> nid=0x3faa runnable [0x00007f84415d4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519937c48> (a sun.nio.ch.Util$2)
> - locked <0x0000000519937c60> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519887d48> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d0d000
> nid=0x3fa9 waiting on condition [0x00007f84416d5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519654310> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913c000
> nid=0x3fa8 waiting on condition [0x00007f84417d6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191b54d8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571164000
> nid=0x3fa7 waiting on condition [0x00007f84418d7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519637a20> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83a801a000
> nid=0x3fa6 runnable [0x00007f84419d8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005195f76c8> (a sun.nio.ch.Util$2)
> - locked <0x00000005196430c0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194ecaf0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83b401a000
> nid=0x3fa5 runnable [0x00007f8441ad9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051965f960> (a sun.nio.ch.Util$2)
> - locked <0x000000051965f978> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519104648> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83b001a000
> nid=0x3fa4 runnable [0x00007f8441bda000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519655e28> (a sun.nio.ch.Util$2)
> - locked <0x0000000519655e40> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519104ba0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-3" prio=10 tid=0x00007f8580052800 nid=0x3fa3 runnable
> [0x00007f8441cdb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519637b10> (a sun.nio.ch.Util$2)
> - locked <0x0000000519694210> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051916e518> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83bc01a000
> nid=0x3fa2 runnable [0x00007f8441ddc000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519597678> (a sun.nio.ch.Util$2)
> - locked <0x0000000519597690> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519106388> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c4800
> nid=0x3fa1 waiting on condition [0x00007f8441edd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519654328> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d0b000
> nid=0x3fa0 waiting on condition [0x00007f8441fde000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519676ad0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571162000
> nid=0x3f9f waiting on condition [0x00007f84420df000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519655618> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913a000
> nid=0x3f9e waiting on condition [0x00007f84421e0000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005194ae248> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-2" prio=10 tid=0x00007f8580001000 nid=0x3f9d runnable
> [0x00007f84422e1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519654340> (a sun.nio.ch.Util$2)
> - locked <0x0000000519654358> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519105380> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83cc01a000
> nid=0x3f9c runnable [0x00007f84423e2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519655e88> (a sun.nio.ch.Util$2)
> - locked <0x0000000519655ea0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519104be8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83d001a000
> nid=0x3f9b runnable [0x00007f84424e3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005194e1350> (a sun.nio.ch.Util$2)
> - locked <0x00000005194e1368> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005191041c8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c3000
> nid=0x3f9a waiting on condition [0x00007f84425e4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051967caf8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83d801a000
> nid=0x3f99 runnable [0x00007f84426e5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519655ee8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519655f00> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519104c30> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83dc01a000
> nid=0x3f98 runnable [0x00007f84427e6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005196543a0> (a sun.nio.ch.Util$2)
> - locked <0x00000005196543b8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005191053c8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d09800
> nid=0x3f97 waiting on condition [0x00007f84428e7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519655630> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579138000
> nid=0x3f96 waiting on condition [0x00007f84429e8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005196592c8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571160000
> nid=0x3f95 waiting on condition [0x00007f8442ae9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519655f48> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f840801a000
> nid=0x3f94 runnable [0x00007f8442bea000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519655648> (a sun.nio.ch.Util$2)
> - locked <0x0000000519655660> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051914b290> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83e801a000
> nid=0x3f93 runnable [0x00007f8442ceb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519643108> (a sun.nio.ch.Util$2)
> - locked <0x0000000519643120> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194ecb38> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d07800
> nid=0x3f92 waiting on condition [0x00007f8442dec000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519658428> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83f801a000
> nid=0x3f91 runnable [0x00007f8442eed000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005194c9908> (a sun.nio.ch.Util$2)
> - locked <0x00000005194c9920> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519511968> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83fc01a000
> nid=0x3f90 runnable [0x00007f8442fee000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519643168> (a sun.nio.ch.Util$2)
> - locked <0x0000000519643180> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194ecb80> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578465000
> nid=0x3f8f waiting on condition [0x00007f8403ffe000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519654400> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857115e800
> nid=0x3f8e waiting on condition [0x00007f84430ef000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005196556a8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c1000
> nid=0x3f8d waiting on condition [0x00007f84431f0000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519654418> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f840c01a000
> nid=0x3f8c runnable [0x00007f84432f1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005196431c8> (a sun.nio.ch.Util$2)
> - locked <0x00000005196431e0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194ecbc8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f841801a000
> nid=0x3f8b runnable [0x00007f84433f2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519655f60> (a sun.nio.ch.Util$2)
> - locked <0x0000000519655f78> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519104c78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b4000
> nid=0x3f8a waiting on condition [0x00007f84434f3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005196721e8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5bf000
> nid=0x3f89 waiting on condition [0x00007f84435f4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519654430> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f841c01a000
> nid=0x3f88 runnable [0x00007f84436f5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005196556c0> (a sun.nio.ch.Util$2)
> - locked <0x00000005196556d8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051914b2d8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f842801a000
> nid=0x3f87 runnable [0x00007f84437f6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519643228> (a sun.nio.ch.Util$2)
> - locked <0x0000000519643240> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194ecc10> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b2000
> nid=0x3f86 waiting on condition [0x00007f84438f7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519654448> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5bd800
> nid=0x3f85 waiting on condition [0x00007f84439f8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051967cb10> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f843001a000
> nid=0x3f84 runnable [0x00007f8443af9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519655fc0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519655fd8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519104cc0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b0800
> nid=0x3f83 waiting on condition [0x00007f8443bfa000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005196592e0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f843801a000
> nid=0x3f82 runnable [0x00007f8443cfb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519643288> (a sun.nio.ch.Util$2)
> - locked <0x00000005196432a0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194ecc58> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d17b000
> nid=0x3f81 waiting on condition [0x00007f8443dfc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051921afd8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f843c01a000
> nid=0x3f80 runnable [0x00007f8443efd000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519654460> (a sun.nio.ch.Util$2)
> - locked <0x0000000519654478> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519105410> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08ae800
> nid=0x3f7f waiting on condition [0x00007f8443ffe000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519676ae8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f844401a000
> nid=0x3f7e runnable [0x00007f85441a6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051967cb28> (a sun.nio.ch.Util$2)
> - locked <0x000000051967cb40> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194745e0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d179000
> nid=0x3f7d waiting on condition [0x00007f85442a7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005196432e8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f845801a000
> nid=0x3f7c runnable [0x00007f85443a8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519658440> (a sun.nio.ch.Util$2)
> - locked <0x00000005196583f8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519349128> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f845401a000
> nid=0x3f7b runnable [0x00007f85444a9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519676aa0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519676ab8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051921b020> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08ad800
> nid=0x3f7a waiting on condition [0x00007f85446c5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519659298> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578463000
> nid=0x3f79 waiting on condition [0x00007f85447c6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005194a3568> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f845c01a000
> nid=0x3f78 runnable [0x00007f85445aa000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005196544c0> (a sun.nio.ch.Util$2)
> - locked <0x00000005196544d8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519105458> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d177000
> nid=0x3f77 waiting on condition [0x00007f85448c7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519656020> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f846401a000
> nid=0x3f76 runnable [0x00007f85449c8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519655720> (a sun.nio.ch.Util$2)
> - locked <0x0000000519655738> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519105f98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857115c800
> nid=0x3f75 waiting on condition [0x00007f8544ac9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519402f58> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f847801a000
> nid=0x3f74 runnable [0x00007f8544bca000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519654520> (a sun.nio.ch.Util$2)
> - locked <0x0000000519654538> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005191054a0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f847401a000
> nid=0x3f73 runnable [0x00007f8544ccb000]
>    java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.newKeyIterator(HashMap.java:840)
> at java.util.HashMap$KeySet.iterator(HashMap.java:874)
>  at java.util.HashSet.iterator(HashSet.java:153)
> at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
>  - locked <0x0000000519656248> (a java.util.HashSet)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519656038> (a sun.nio.ch.Util$2)
>  - locked <0x0000000519656050> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519104d08> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0015800
> nid=0x3f72 waiting on condition [0x00007f8544dcc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>  at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051941d010> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578461000
> nid=0x3f71 waiting on condition [0x00007f8544ecd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519477938> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f847c01a000
> nid=0x3f70 runnable [0x00007f8544fce000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051942aa08> (a sun.nio.ch.Util$2)
> - locked <0x000000051942aa20> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519349050> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d175800
> nid=0x3f6f waiting on condition [0x00007f85450cf000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519470c60> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f848401a000
> nid=0x3f6e runnable [0x00007f85451d0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519445fa0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519445fb8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194679d0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570456800
> nid=0x3f6d waiting on condition [0x00007f85452d1000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519419230> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f849801a000
> nid=0x3f6c runnable [0x00007f85453d2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519470c78> (a sun.nio.ch.Util$2)
> - locked <0x0000000519470c90> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194743a0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f849401a000
> nid=0x3f6b runnable [0x00007f85454d3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051945beb8> (a sun.nio.ch.Util$2)
> - locked <0x000000051945bed0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519196000> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0013800
> nid=0x3f6a waiting on condition [0x00007f85455d4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519470cd8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857845f000
> nid=0x3f69 waiting on condition [0x00007f85456d5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051941d1c0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f849c01a000
> nid=0x3f68 runnable [0x00007f85457d6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005194583b8> (a sun.nio.ch.Util$2)
> - locked <0x00000005194583d0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194743e8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c488000
> nid=0x3f67 waiting on condition [0x00007f85458d7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519424e80> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84a401a000
> nid=0x3f66 runnable [0x00007f85459d8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051941d1d8> (a sun.nio.ch.Util$2)
> - locked <0x000000051941d1f0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519467a18> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570455000
> nid=0x3f65 waiting on condition [0x00007f8545ad9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051945bf18> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84b801a000
> nid=0x3f64 runnable [0x00007f8545bda000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519419248> (a sun.nio.ch.Util$2)
> - locked <0x0000000519419200> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519431ee0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84c001a000
> nid=0x3f63 runnable [0x00007f8545cdb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051941d238> (a sun.nio.ch.Util$2)
> - locked <0x000000051941d250> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519467a60> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0012000
> nid=0x3f62 waiting on condition [0x00007f8545ddc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519458418> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84bc01a000
> nid=0x3f61 runnable [0x00007f8545edd000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051945bf30> (a sun.nio.ch.Util$2)
> - locked <0x000000051945bf48> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519196048> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857845e000
> nid=0x3f60 waiting on condition [0x00007f8545fde000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519479f10> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c486000
> nid=0x3f5f waiting on condition [0x00007f85460df000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051941d298> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84c401a000
> nid=0x3f5e runnable [0x00007f85461e0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051942a9a8> (a sun.nio.ch.Util$2)
> - locked <0x000000051942a9c0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519349098> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570453800
> nid=0x3f5d waiting on condition [0x00007f85462e1000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051945bf90> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84cc01a000
> nid=0x3f5c runnable [0x00007f85463e2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519458430> (a sun.nio.ch.Util$2)
> - locked <0x0000000519458448> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519474430> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c484800
> nid=0x3f5b waiting on condition [0x00007f85464e3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051941d2b0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84d401a000
> nid=0x3f5a runnable [0x00007f85465e4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005194191e8> (a sun.nio.ch.Util$2)
> - locked <0x000000051936a4e0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519431f28> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570450800
> nid=0x3f59 waiting on condition [0x00007f85466e5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051943ef60> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84dc01a000
> nid=0x3f58 runnable [0x00007f85467e6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051942a948> (a sun.nio.ch.Util$2)
> - locked <0x000000051942a960> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005193490e0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c482800
> nid=0x3f57 waiting on condition [0x00007f85468e7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051945bfa8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84e401a000
> nid=0x3f56 runnable [0x00007f85469e8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051941d2c8> (a sun.nio.ch.Util$2)
> - locked <0x000000051941d2e0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519467aa8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857044f000
> nid=0x3f55 waiting on condition [0x00007f8546ae9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051945bfc0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84f801a000
> nid=0x3f54 runnable [0x00007f8546bea000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051936a4c8> (a sun.nio.ch.Util$2)
> - locked <0x000000051936a480> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519431f70> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84f401a000
> nid=0x3f53 runnable [0x00007f8546ceb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519458490> (a sun.nio.ch.Util$2)
> - locked <0x00000005194584a8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519474478> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578008000
> nid=0x3f52 waiting on condition [0x00007f8546dec000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051942a8e8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c00d000
> nid=0x3f51 waiting on condition [0x00007f8546eed000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519447098> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84fc01a000
> nid=0x3f50 runnable [0x00007f8546fee000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051941d328> (a sun.nio.ch.Util$2)
> - locked <0x000000051941d340> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519467af0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857044d000
> nid=0x3f4f waiting on condition [0x00007f85470ef000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051945bfd8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f851001a000
> nid=0x3f4e runnable [0x00007f85471f0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519424e98> (a sun.nio.ch.Util$2)
> - locked <0x0000000519424e50> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519380d20> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f850c01a000
> nid=0x3f4d runnable [0x00007f85472f1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005194584f0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519458508> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194744c0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578006800
> nid=0x3f4c waiting on condition [0x00007f85473f2000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051945bff0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c01d800
> nid=0x3f4b waiting on condition [0x00007f85474f3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051936a468> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f851401a000
> nid=0x3f4a runnable [0x00007f85475f4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051941d388> (a sun.nio.ch.Util$2)
> - locked <0x000000051941d3a0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519467b38> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857045b800
> nid=0x3f49 waiting on condition [0x00007f85476f5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051942a900> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f853801a000
> nid=0x3f48 runnable [0x00007f85477f6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519479f28> (a sun.nio.ch.Util$2)
> - locked <0x0000000519479ee0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519203068> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f853001f800
> nid=0x3f47 runnable [0x00007f85478f7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519477950> (a sun.nio.ch.Util$2)
> - locked <0x0000000519477968> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051922faf0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f852c01f000
> nid=0x3f46 runnable [0x00007f85479f8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005194353f8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519435410> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051921b0f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f853401a000
> nid=0x3f45 runnable [0x00007f8547af9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051945c008> (a sun.nio.ch.Util$2)
> - locked <0x000000051945c020> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519196090> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578005800
> nid=0x3f44 waiting on condition [0x00007f8547bfa000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005190de788> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f001a800
> nid=0x3f43 waiting on condition [0x00007f8547cfb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518fae040> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c022000
> nid=0x3f42 waiting on condition [0x00007f8547dfc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518cafb20> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570460000
> nid=0x3f41 waiting on condition [0x00007f8547efd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005190db9c8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-2" prio=10 tid=0x00007f8588036000 nid=0x3f40 runnable
> [0x00007f8547ffe000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005190ee0e8> (a sun.nio.ch.Util$2)
> - locked <0x00000005191f0be8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005190e1d18> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-1" prio=10 tid=0x00007f858801b800 nid=0x3f3f runnable
> [0x00007f858c18a000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000518f16658> (a sun.nio.ch.Util$2)
> - locked <0x000000051941f820> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005190edc80> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-1" prio=10 tid=0x00007f8580002800 nid=0x3f3e runnable
> [0x00007f858c28b000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051913eb50> (a sun.nio.ch.Util$2)
> - locked <0x000000051913eca0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000518c9c4e8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-148-thread-1" prio=10 tid=0x00007f856844c000 nid=0x3f3d waiting on
> condition [0x00007f858c38c000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005190cfe98> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-146-thread-1" prio=10 tid=0x00007f85dc006800 nid=0x3f3c waiting on
> condition [0x00007f858c48d000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for  <0x0000000519018558> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>  at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-147-thread-1" prio=10 tid=0x00007f85e82b1000 nid=0x3f3b waiting on
> condition [0x00007f858c58e000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051903b488> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-145-thread-1" prio=10 tid=0x00007f8574027000 nid=0x3f3a waiting on
> condition [0x00007f858c68f000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for  <0x000000051903b998> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>  at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-141-thread-1" prio=10 tid=0x00007f85e8159000 nid=0x3f39 runnable
> [0x00007f861cd8e000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005190db9e0> (a sun.nio.ch.Util$2)
> - locked <0x00000005190db998> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000518c9c020> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-138-thread-1" prio=10 tid=0x00007f856844a800 nid=0x3f38 runnable
> [0x00007f861cf90000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005190e1b68> (a sun.nio.ch.Util$2)
> - locked <0x00000005190e1b80> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000518c9c608> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-143-thread-1" prio=10 tid=0x00007f85dc003000 nid=0x3f37 runnable
> [0x00007f861d091000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519018fa8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519018fc0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005190e2228> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-137-thread-1" prio=10 tid=0x00007f8574007000 nid=0x3f36 runnable
> [0x00007f861ce8f000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519003dc8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519003de0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005190ea1a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "1407844613@qtp-1263282395-1 - Acceptor0 SocketConnector@0.0.0.0:54321"
> prio=10 tid=0x00007f85e404a800 nid=0x3f35 runnable [0x00007f858c790000]
>    java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
>  - locked <0x00000005190e2720> (a java.net.SocksSocketImpl)
> at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>  at java.net.ServerSocket.accept(ServerSocket.java:430)
> at org.mortbay.jetty.bio.SocketConnector.accept(SocketConnector.java:99)
>  at
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>
> "208924888@qtp-1263282395-0" prio=10 tid=0x00007f85e4021800 nid=0x3f34 in
> Object.wait() [0x00007f861c15b000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>  at java.lang.Object.wait(Native Method)
>  - waiting on <0x00000005190d0528> (a
> org.mortbay.thread.QueuedThreadPool$PoolThread)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
>  - locked <0x00000005190d0528> (a
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>
> "lifecycleSupervisor-1-9" prio=10 tid=0x00007f85e401a800 nid=0x3f33
> waiting on condition [0x00007f861c687000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518c85670> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> "lifecycleSupervisor-1-8" prio=10 tid=0x00007f85e4018800 nid=0x3f32
> waiting on condition [0x00007f861c788000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for  <0x0000000518c85670> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>  at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "lifecycleSupervisor-1-7" prio=10 tid=0x00007f85e4016800 nid=0x3f31
> waiting on condition [0x00007f861c889000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518c85670> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> "lifecycleSupervisor-1-6" prio=10 tid=0x00007f85e4014800 nid=0x3f30
> waiting on condition [0x00007f861c98a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for  <0x0000000518c85670> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>  at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "lifecycleSupervisor-1-5" prio=10 tid=0x00007f85e4013000 nid=0x3f2f
> waiting on condition [0x00007f861ca8b000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518c85670> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> "lifecycleSupervisor-1-4" prio=10 tid=0x00007f85e4028000 nid=0x3f2e
> waiting on condition [0x00007f861cb8c000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for  <0x0000000518c85670> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>  at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "lifecycleSupervisor-1-3" prio=10 tid=0x00007f85e4026800 nid=0x3f2d
> waiting on condition [0x00007f861cc8d000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518c85670> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> "Log-BackgroundWorker-ch1" prio=10 tid=0x00007f85dc001000 nid=0x3f27
> waiting on condition [0x00007f861d299000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for  <0x0000000519017718> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>  at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Log-BackgroundWorker-ch2" prio=10 tid=0x00007f85e80ca000 nid=0x3f26
> waiting on condition [0x00007f861d39a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051903b4e8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> "lifecycleSupervisor-1-2" prio=10 tid=0x00007f85e4025000 nid=0x3f25
> waiting on condition [0x00007f861d6ba000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for  <0x0000000518c85670> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>  at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "lifecycleSupervisor-1-1" prio=10 tid=0x00007f85e400e800 nid=0x3f24
> waiting on condition [0x00007f861d7bb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518c85670> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> "conf-file-poller-0" prio=10 tid=0x00007f85f0001000 nid=0x3f23 waiting on
> condition [0x00007f861d8bc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for  <0x0000000518ec72f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>  at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "DestroyJavaVM" prio=10 tid=0x00007f86a800c800 nid=0x3efa waiting on
> condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "lifecycleSupervisor-1-0" prio=10 tid=0x00007f86a89f5800 nid=0x3f22
> waiting on condition [0x00007f861d9bd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>  at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518c85670> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> "Low Memory Detector" daemon prio=10 tid=0x00007f86a831d800 nid=0x3f20
> runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" daemon prio=10 tid=0x00007f86a831b000 nid=0x3f1f
> waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" daemon prio=10 tid=0x00007f86a8319000 nid=0x3f1e
> waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x00007f86a8316800 nid=0x3f1d
> runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (Concurrent GC)" daemon prio=10
> tid=0x00007f86a8314800 nid=0x3f1c waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x00007f86a82f9800 nid=0x3f1b in
> Object.wait() [0x00007f861e3eb000]
>    java.lang.Thread.State: WAITING (on object monitor)
>  at java.lang.Object.wait(Native Method)
> - waiting on <0x0000000518c843c0> (a java.lang.ref.ReferenceQueue$Lock)
>  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> - locked <0x0000000518c843c0> (a java.lang.ref.ReferenceQueue$Lock)
>  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=0x00007f86a82f7800 nid=0x3f1a in
> Object.wait() [0x00007f861e4ec000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
>  - waiting on <0x0000000518c80860> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:485)
>  at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> - locked <0x0000000518c80860> (a java.lang.ref.Reference$Lock)
>
> "VM Thread" prio=10 tid=0x00007f86a82f1000 nid=0x3f19 runnable
>
> "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f86a801b800
> nid=0x3efb runnable
>
> "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f86a801d000
> nid=0x3efc runnable
>
> "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f86a801f000
> nid=0x3efd runnable
>
> "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f86a8021000
> nid=0x3efe runnable
>
> "Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x00007f86a8022800
> nid=0x3eff runnable
>
> "Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x00007f86a8024800
> nid=0x3f00 runnable
>
> "Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x00007f86a8026000
> nid=0x3f01 runnable
>
> "Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x00007f86a8028000
> nid=0x3f02 runnable
>
> "Gang worker#8 (Parallel GC Threads)" prio=10 tid=0x00007f86a802a000
> nid=0x3f03 runnable
>
> "Gang worker#9 (Parallel GC Threads)" prio=10 tid=0x00007f86a802b800
> nid=0x3f04 runnable
>
> "Gang worker#10 (Parallel GC Threads)" prio=10 tid=0x00007f86a802d800
> nid=0x3f05 runnable
>
> "Gang worker#11 (Parallel GC Threads)" prio=10 tid=0x00007f86a802f800
> nid=0x3f06 runnable
>
> "Gang worker#12 (Parallel GC Threads)" prio=10 tid=0x00007f86a8031000
> nid=0x3f07 runnable
>
> "Gang worker#13 (Parallel GC Threads)" prio=10 tid=0x00007f86a8033000
> nid=0x3f08 runnable
>
> "Gang worker#14 (Parallel GC Threads)" prio=10 tid=0x00007f86a8035000
> nid=0x3f09 runnable
>
> "Gang worker#15 (Parallel GC Threads)" prio=10 tid=0x00007f86a8036800
> nid=0x3f0a runnable
>
> "Gang worker#16 (Parallel GC Threads)" prio=10 tid=0x00007f86a8038800
> nid=0x3f0b runnable
>
> "Gang worker#17 (Parallel GC Threads)" prio=10 tid=0x00007f86a803a000
> nid=0x3f0c runnable
>
> "Gang worker#18 (Parallel GC Threads)" prio=10 tid=0x00007f86a803c000
> nid=0x3f0d runnable
>
> "Gang worker#19 (Parallel GC Threads)" prio=10 tid=0x00007f86a803e000
> nid=0x3f0e runnable
>
> "Gang worker#20 (Parallel GC Threads)" prio=10 tid=0x00007f86a803f800
> nid=0x3f0f runnable
>
> "Gang worker#21 (Parallel GC Threads)" prio=10 tid=0x00007f86a8041800
> nid=0x3f10 runnable
>
> "Gang worker#22 (Parallel GC Threads)" prio=10 tid=0x00007f86a8043800
> nid=0x3f11 runnable
>
> "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f86a82aa000
> nid=0x3f18 runnable
> "Gang worker#0 (Parallel CMS Threads)" prio=10 tid=0x00007f86a829e800
> nid=0x3f12 runnable
>
> "Gang worker#1 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a0000
> nid=0x3f13 runnable
>
> "Gang worker#2 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a2000
> nid=0x3f14 runnable
>
> "Gang worker#3 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a3800
> nid=0x3f15 runnable
>
> "Gang worker#4 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a5800
> nid=0x3f16 runnable
>
> "Gang worker#5 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a7800
> nid=0x3f17 runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x00007f86a8329000 nid=0x3f21
> waiting on condition
>
> JNI global references: 1749
>
>
>
> --
>
>
> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com
>
> Pankaj Gupta | Software Engineer
>
> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>
>
> United States | Canada | United Kingdom | Germany
>
>
> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
> !
>
>
>
>
>
> --
>
>
> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com
>
> Pankaj Gupta | Software Engineer
>
> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>
>
> United States | Canada | United Kingdom | Germany
>
>
> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
> !
>
>
>


-- 
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org

Re: Lock contention in FileChannel

[Hari Shreedharan <hs...@cloudera.com>]

Yes, IO is done inside locks to avoid multiple takes and puts getting written out at the same time. Even though Java makes sure the writes are serialized, Flume still needs to keep track of some counters etc, so the lock is required. Note that the lock you are talking about is  in the LogFile class, which represents a single file - so even if the write is inside that lock (which is also inside that class itself) that  does not cause any contention - because the lock is just preventing two IO ops to happen at the same time. 


Thanks,
Hari


On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:

> It seems like some i/o is done inside the lock, which means that time for taking a lock is proportional to the time for i/o and thus it becomes a problem. I apologize in advance if I am wrong but the call stack and behavior I'm seeing seems to suggest that. Specifically, it seems that we do a write while inside take:
> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10 tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.NativeThread.current(Native Method)
>         at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>         at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>         - locked <0x00000005190ec998> (a java.lang.Object)
>         at org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>         at org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>         - locked <0x0000000519111590> (a org.apache.flume.channel.file.LogFileV3$Writer)
>         at org.apache.flume.channel.file.Log.take(Log.java:550)
>         at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>         at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>         at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>         at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>         at org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:662)
> 
> 
> 
> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <hshreedharan@cloudera.com (mailto:hshreedharan@cloudera.com)> wrote:
> > Since the channel is designed to make sure that events are not duplicated to multiple sinks, and to protect against corruption due to concurrency issues, we do not need the locking in the channel's flume event queue. It is unlikely that locking is what is causing performance issues because the channel is heavily I/O bound. If you take a series of thread dumps, you will probably see that those threads are moving forward and the ones reading/writing from/to disk are the ones which are slower. These locks are unlikely to hit performance much. 
> > 
> > Thanks,
> > Hari
> > 
> > 
> > On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
> > 
> > > Hi,
> > > 
> > > Spent some more time debugging issues with FileChannel. The problem seems to lock contention reading from FlumeEventQueue:
> > > 
> > > I see a lot of threads like this: 
> > > "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10 tid=0x00007f857b378800 nid=0x404d waiting for monitor entry [0x00007f821afee000]
> > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > >         at org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
> > >         - waiting to lock <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> > >         at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
> > >         at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> > >         at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> > >         at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
> > >         at org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> > >         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> > >         at java.lang.Thread.run(Thread.java:662)
> > > 
> > > 
> > > I have two file channels and 8 Avro Sinks per file channel. I added more sinks because they weren't draining fast enough. It seems like they send the batch then wait for an ack before sending again, thus sends are not pipelined and having more sinks seemed like a good way of getting some parallelism. 
> > > 
> > > Here's the full stack trace:
> > > 2013-08-13 15:30:32
> > > Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02 mixed mode):
> > > 
> > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000] 
> > >    java.lang.Thread.State: WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000056d932120> (a java.util.concurrent.CountDownLatch$Sync)
> > > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> > > at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
> > > at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
> > > at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
> > > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
> > > at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
> > > at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
> > > at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
> > > at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
> > > at $Proxy7.appendBatch(Unknown Source)
> > > at org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
> > > at org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
> > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Attach Listener" daemon prio=10 tid=0x00007f85fc016000 nid=0x41a1 waiting on condition [0x0000000000000000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > 
> > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005194c3340> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005191a16d8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519174e08> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519174e98> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005195f9670> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519174ce8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005191a1918> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005191e7f60> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153 runnable [0x00007f8210e4d000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
> > > - locked <0x000000054a2917a8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519174f58> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146 runnable [0x00007f8210f4e000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000546e5dd28> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118 runnable [0x00007f821104f000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000544f763c0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111 runnable [0x00007f8211150000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
> > > - locked <0x000000054473d5a8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110 runnable [0x00007f8211251000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005437f4f70> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f runnable [0x00007f8211352000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005422a9018> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e runnable [0x00007f8211453000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000541063d50> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d runnable [0x00007f8211554000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000540dc1110> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a runnable [0x00007f8211655000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000053fbbc3b0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109 runnable [0x00007f8211756000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
> > > - locked <0x000000053f812860> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd runnable [0x00007f8211857000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
> > > - locked <0x000000053e6e5570> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519175138> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519174c28> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005191751c8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0 runnable [0x00007f8211c5b000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000536579db0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005191750a8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1 runnable [0x00007f8211e5d000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005313943a8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051936f460> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051936dfa8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de runnable [0x00007f8212160000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
> > > - locked <0x000000052dad4468> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd runnable [0x00007f8212261000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
> > > - locked <0x000000052c73c5f0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc runnable [0x00007f8212362000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
> > > - locked <0x000000052b8da998> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db waiting for monitor entry [0x00007f8212462000] 
> > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > at org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
> > > - waiting to lock <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
> > > at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> > > at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> > > at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> > > at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> > > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> > > at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051936df18> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000518fe7ae8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8 runnable [0x00007f8212766000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005263d6ba8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7 runnable [0x00007f8212867000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000525d1d0e0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6 runnable [0x00007f8212968000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000525a689d8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5 runnable [0x00007f8212a69000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000524e1e7d8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4 runnable [0x00007f8212b6a000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000524d437a8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-36" prio=10 tid=0x00007f85843ad800 nid=0x40d3 runnable [0x00007f8212c6b000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000524cf4140> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000524cf4128> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000524ab32e8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-35" prio=10 tid=0x00007f85d83a9000 nid=0x40d2 runnable [0x00007f8212d6c000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005244877c8> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005244877b0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000523f5b660> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-34" prio=10 tid=0x00007f85d838e000 nid=0x40d1 runnable [0x00007f8212e6d000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000052419e490> (a sun.nio.ch.Util$2)
> > > - locked <0x000000052419e478> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000523f5b6f0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-34" prio=10 tid=0x00007f8588390800 nid=0x40d0 waiting for monitor entry [0x00007f8212f6d000] 
> > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
> > > - waiting to lock <0x0000000519111590> (a org.apache.flume.channel.file.LogFileV3$Writer)
> > > at org.apache.flume.channel.file.Log.put(Log.java:507)
> > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
> > > at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> > > at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> > > at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> > > at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> > > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> > > at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-35" prio=10 tid=0x00007f85843ab800 nid=0x40cf runnable [0x00007f821306f000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000523d40938> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000523d40aa0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000523ecec18> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-37" prio=10 tid=0x00007f85803e3800 nid=0x40ce waiting for monitor entry [0x00007f821316f000] 
> > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > at org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
> > > - waiting to lock <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
> > > at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> > > at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> > > at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> > > at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> > > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> > > at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-34" prio=10 tid=0x00007f8584391000 nid=0x40cd runnable [0x00007f8213271000]
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000523935540> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000523935528> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000523788ed8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-33" prio=10 tid=0x00007f85d8373800 nid=0x40cc runnable [0x00007f8213372000]
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005239ec658> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005239ec640> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000523845410> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-33" prio=10 tid=0x00007f8588375800 nid=0x40cb waiting for monitor entry [0x00007f8213472000]
> > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
> > > - waiting to lock <0x0000000519111590> (a org.apache.flume.channel.file.LogFileV3$Writer)
> > > at org.apache.flume.channel.file.Log.put(Log.java:507)
> > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
> > > at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> > > at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> > > at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> > > at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> > > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> > > at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-33" prio=10 tid=0x00007f8584376000 nid=0x40ca runnable [0x00007f8213574000]
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005237e3028> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005237e3010> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000523788e90> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-32" prio=10 tid=0x00007f85d8358800 nid=0x40c9 waiting for monitor entry [0x00007f8213674000]
> > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > at org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
> > > - waiting to lock <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
> > > at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> > > at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> > > at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> > > at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> > > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> > > at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-32" prio=10 tid=0x00007f858835b000 nid=0x40c8 runnable [0x00007f8213776000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000523343688> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000523343670> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000052305e5c8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-36" prio=10 tid=0x00007f85803c9000 nid=0x40c7 runnable [0x00007f8213877000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005230fa580> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005230fa568> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000522f5ca50> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-32" prio=10 tid=0x00007f858435b000 nid=0x40c6 runnable [0x00007f8213978000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005231367a8> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000523136790> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000052305e658> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-31" prio=10 tid=0x00007f85d833e000 nid=0x40c5 runnable [0x00007f8213a79000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000522db30a0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000522db3088> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000522bda430> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-31" prio=10 tid=0x00007f8588340000 nid=0x40c4 runnable [0x00007f8213b7a000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000522c16518> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000522c16500> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000522bd97e8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-30" prio=10 tid=0x00007f85d8323800 nid=0x40c3 runnable [0x00007f8213c7b000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000522bd1a20> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000522bd1a08> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000522bda3a0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-31" prio=10 tid=0x00007f8584340800 nid=0x40c2 runnable [0x00007f8213d7c000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005228d1608> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005228d15f0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005228a6aa0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-30" prio=10 tid=0x00007f8588325000 nid=0x40c1 runnable [0x00007f8213e7d000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000052277b088> (a sun.nio.ch.Util$2)
> > > - locked <0x000000052277b070> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005225ed3b8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-35" prio=10 tid=0x00007f85803ae000 nid=0x40c0 runnable [0x00007f8213f7e000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000522737380> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000522737368> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005225ed328> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-29" prio=10 tid=0x00007f85d8308800 nid=0x40bf runnable [0x00007f821407f000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005226034a8> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000522603490> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005225ed298> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-28" prio=10 tid=0x00007f85d82ed800 nid=0x40be runnable [0x00007f8214180000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000522359408> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005223593f0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005221e7278> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-27" prio=10 tid=0x00007f85d82d2800 nid=0x40bd runnable [0x00007f8214281000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005222c0478> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005222c0460> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005221e6e40> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-29" prio=10 tid=0x00007f858830a800 nid=0x40bc runnable [0x00007f8214382000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000521ddf488> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000521ddf470> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000521c88d38> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-30" prio=10 tid=0x00007f8584326000 nid=0x40bb waiting for monitor entry [0x00007f8214482000] 
> > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
> > > - waiting to lock <0x0000000519111590> (a org.apache.flume.channel.file.LogFileV3$Writer)
> > > at org.apache.flume.channel.file.Log.put(Log.java:507)
> > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
> > > at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> > > at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> > > at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> > > at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> > > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> > > at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-26" prio=10 tid=0x00007f85d82b8000 nid=0x40ba runnable [0x00007f8214584000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000521cbcb90> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000521cbcb78> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000521279250> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-28" prio=10 tid=0x00007f85882ef800 nid=0x40b9 runnable [0x00007f8214685000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000521a1c2d0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000521a1c2b8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005218f1508> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-34" prio=10 tid=0x00007f8580393800 nid=0x40b8 runnable [0x00007f8214786000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000052181ce58> (a sun.nio.ch.Util$2)
> > > - locked <0x000000052181ce40> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005212791c0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-29" prio=10 tid=0x00007f858430b800 nid=0x40b7 runnable [0x00007f8214887000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005219175f0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000521746090> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000521278e60> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-27" prio=10 tid=0x00007f85882d4800 nid=0x40b6 runnable [0x00007f8214988000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000052163e9c8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000052163e9b0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000521279130> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-33" prio=10 tid=0x00007f8580378800 nid=0x40b5 runnable [0x00007f8214a89000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000052152cb38> (a sun.nio.ch.Util$2)
> > > - locked <0x000000052152cb20> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000521278d40> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-28" prio=10 tid=0x00007f85842f0800 nid=0x40b4 runnable [0x00007f8214b8a000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005213ba3e0> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005213ba3c8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000521278020> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-27" prio=10 tid=0x00007f85842d5800 nid=0x40b3 runnable [0x00007f8214c8b000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005213ed660> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005213ed648> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000521278c20> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-26" prio=10 tid=0x00007f85882b9800 nid=0x40b2 runnable [0x00007f8214d8c000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000052133b7c0> (a sun.nio.ch.Util$2)
> > > - locked <0x000000052133b7a8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005212790a0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-32" prio=10 tid=0x00007f858035d800 nid=0x40af runnable [0x00007f8214e8d000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000520e53be0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000520e53bc8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000520f3bb50> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-25" prio=10 tid=0x00007f85d829d000 nid=0x40ae runnable [0x00007f8214f8e000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000520e09a30> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000520e09a18> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000520f3b6d0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-31" prio=10 tid=0x00007f8580342800 nid=0x40ad runnable [0x00007f821508f000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000520ef70e0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000520ef70c8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000520f3b208> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-26" prio=10 tid=0x00007f85842ba800 nid=0x40ac runnable [0x00007f8215190000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000520c9dfc0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000520c9dfa8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000520bb3cc0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-25" prio=10 tid=0x00007f85842a0000 nid=0x40ab runnable [0x00007f8215291000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000520b941a8> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000520b94190> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000520bb30c0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-24" prio=10 tid=0x00007f85d8282800 nid=0x40aa runnable [0x00007f8215392000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000520952288> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000520952270> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000520356888> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-30" prio=10 tid=0x00007f8580328000 nid=0x40a9 runnable [0x00007f8215493000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000520a2fa20> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000520a2fa08> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000520983da0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-23" prio=10 tid=0x00007f85d8267800 nid=0x40a8 runnable [0x00007f8215594000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000052095c470> (a sun.nio.ch.Util$2)
> > > - locked <0x000000052095c458> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000520356840> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-29" prio=10 tid=0x00007f858030d000 nid=0x40a7 runnable [0x00007f8215695000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000052065b178> (a sun.nio.ch.Util$2)
> > > - locked <0x000000052065b160> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000520357320> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-28" prio=10 tid=0x00007f85802f2800 nid=0x40a6 runnable [0x00007f8215796000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000052043cad0> (a sun.nio.ch.Util$2)
> > > - locked <0x000000052043cab8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005203567b0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-25" prio=10 tid=0x00007f858829f000 nid=0x40a5 runnable [0x00007f8215897000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000052041a8a8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000052041a890> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000520356690> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-22" prio=10 tid=0x00007f85d824c800 nid=0x40a4 runnable [0x00007f8215998000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000052036b380> (a sun.nio.ch.Util$2)
> > > - locked <0x000000052036b368> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000520357290> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-24" prio=10 tid=0x00007f8584285800 nid=0x40a3 runnable [0x00007f8215a99000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000052007ac40> (a sun.nio.ch.Util$2)
> > > - locked <0x000000052007ac28> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051fe41948> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-27" prio=10 tid=0x00007f85802d8000 nid=0x40a2 runnable [0x00007f8215b9a000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005201a2ae8> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005201a2ad0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051ffda458> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-21" prio=10 tid=0x00007f85d8232000 nid=0x40a1 waiting for monitor entry [0x00007f8215c9a000] 
> > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > at org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
> > > - waiting to lock <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
> > > at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> > > at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> > > at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> > > at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> > > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> > > at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-26" prio=10 tid=0x00007f85802bd800 nid=0x40a0 runnable [0x00007f8215d9c000]
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051ff20e78> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051ff20e60> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051fe41900> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-24" prio=10 tid=0x00007f8588284000 nid=0x409f runnable [0x00007f8215e9d000]
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051fbc1ca0> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051fbc1c88> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051f9a7978> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-25" prio=10 tid=0x00007f85802a2800 nid=0x409e runnable [0x00007f8215f9e000]
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051fa3b5b8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051fa3b5a0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051f3155b0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-20" prio=10 tid=0x00007f85d8217800 nid=0x409d runnable [0x00007f821609f000]
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051f83c838> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051f83c820> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051f3150e8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-23" prio=10 tid=0x00007f8588269800 nid=0x409c runnable [0x00007f82161a0000]
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051f78b9b0> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051f78b998> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051f3156d0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-22" prio=10 tid=0x00007f858824e800 nid=0x409b runnable [0x00007f82162a1000]
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051f7c0cc8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051f7c0cb0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051f3151c0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-23" prio=10 tid=0x00007f858426a800 nid=0x409a runnable [0x00007f82163a2000]
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051f6065a8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051f606590> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051f3d9668> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-22" prio=10 tid=0x00007f858424f800 nid=0x4099 runnable [0x00007f82164a3000]
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051f56d710> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051f56d6f8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051f314f80> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-21" prio=10 tid=0x00007f8588233800 nid=0x4098 runnable [0x00007f82165a4000]
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051f3bcd68> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051f3bcd50> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051f315400> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-19" prio=10 tid=0x00007f85d81fc800 nid=0x4097 runnable [0x00007f82166a5000]
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051f53b198> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051f53b180> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051f315760> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-20" prio=10 tid=0x00007f8588218800 nid=0x4096 waiting for monitor entry [0x00007f82167a5000]
> > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
> > > - waiting to lock <0x0000000519111590> (a org.apache.flume.channel.file.LogFileV3$Writer)
> > > at org.apache.flume.channel.file.Log.put(Log.java:507)
> > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
> > > at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> > > at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> > > at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> > > at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> > > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> > > at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-18" prio=10 tid=0x00007f85d81e1800 nid=0x4095 runnable [0x00007f82168a7000]
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051ee92ba0> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051ee92b88> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051ed19740> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-19" prio=10 tid=0x00007f85881fd800 nid=0x4094 runnable [0x00007f82169a8000]
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051ee5da40> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051ee5da28> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051e77a940> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-17" prio=10 tid=0x00007f85d81c6800 nid=0x4093 runnable [0x00007f8216aa9000]
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051ecb1730> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051ecb1718> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051e77a8b0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-21" prio=10 tid=0x00007f8584234800 nid=0x4092 runnable [0x00007f8216baa000]
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051e899660> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051e899648> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051e77a310> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-24" prio=10 tid=0x00007f8580287000 nid=0x4091 runnable [0x00007f8216caa000]
> > >    java.lang.Thread.State: RUNNABLE
> > > at org.apache.flume.channel.file.FlumeEventQueue.addTail(FlumeEventQueue.java:165)
> > > - locked <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:529)
> > > - locked <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> > > at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
> > > at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
> > > at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> > > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> > > at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-23" prio=10 tid=0x00007f858026c000 nid=0x4090 runnable [0x00007f8216dac000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051ea8b6f0> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051ea8b6d8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051e77a820> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-16" prio=10 tid=0x00007f85d81ac000 nid=0x408f runnable [0x00007f8216ead000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051e7c8808> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051e7c87f0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051e77a280> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-20" prio=10 tid=0x00007f8584219800 nid=0x408e runnable [0x00007f8216fae000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051e8fffa8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051e8fff90> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051e77a700> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-22" prio=10 tid=0x00007f8580251000 nid=0x408d runnable [0x00007f82170af000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051e4dc398> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051e4dc380> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051dcbeef8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-19" prio=10 tid=0x00007f85841fe800 nid=0x408c runnable [0x00007f82171b0000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051e221520> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051e221508> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051e407860> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-18" prio=10 tid=0x00007f85841e3800 nid=0x408b runnable [0x00007f82172b1000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051e465298> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051e465280> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051dcbef88> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-18" prio=10 tid=0x00007f85881e3000 nid=0x408a runnable [0x00007f82173b2000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051dea12e0> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051dea12c8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051cfd4d78> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-15" prio=10 tid=0x00007f85d8191000 nid=0x4089 runnable [0x00007f82174b3000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051de322b8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051de322a0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051dcbec28> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-14" prio=10 tid=0x00007f85d8176800 nid=0x4088 runnable [0x00007f82175b4000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051dea29a8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051dea2990> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051dcbf018> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-21" prio=10 tid=0x00007f8580236800 nid=0x4087 runnable [0x00007f82176b5000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051dca95d0> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051dca95b8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051cfd4e08> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-17" prio=10 tid=0x00007f85841c8800 nid=0x4086 runnable [0x00007f82177b6000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051da8d058> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051da8d040> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051cd693f8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-20" prio=10 tid=0x00007f858021b800 nid=0x4085 runnable [0x00007f82178b7000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051d95a5b0> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051d95a598> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051cfd4e98> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-16" prio=10 tid=0x00007f85841ad800 nid=0x4083 runnable [0x00007f82179b8000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051d92a278> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051d92a260> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051cfd4fb8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-13" prio=10 tid=0x00007f85d815b800 nid=0x4082 runnable [0x00007f8217ab9000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051d734788> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051d734770> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051cd69488> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-19" prio=10 tid=0x00007f8580200800 nid=0x4081 runnable [0x00007f8217bba000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051d4268e8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051d4268d0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051d139810> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-15" prio=10 tid=0x00007f8584192800 nid=0x4080 runnable [0x00007f8217cbb000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051d4cda88> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051d4cda70> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051babcf98> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-17" prio=10 tid=0x00007f85881c8000 nid=0x407f runnable [0x00007f8217dbc000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051d32ffe8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051d32ffd0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051cd691b8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-14" prio=10 tid=0x00007f8584177800 nid=0x407e runnable [0x00007f8217ebd000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051d506c60> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051d506c48> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051cfd5048> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-16" prio=10 tid=0x00007f85881ad000 nid=0x407d runnable [0x00007f8217fbe000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051d1223e0> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051d1223c8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051c6c7b40> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-15" prio=10 tid=0x00007f8588192800 nid=0x407c runnable [0x00007f82180bf000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051d146f50> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051d146f38> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051cd69248> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-13" prio=10 tid=0x00007f858415c800 nid=0x407b runnable [0x00007f82181c0000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051d11d1b8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051d11d1a0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051c6c7c18> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-12" prio=10 tid=0x00007f85d8141000 nid=0x407a runnable [0x00007f82182c1000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051cd01a78> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051cd01a60> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051cd695a8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-11" prio=10 tid=0x00007f85d8126800 nid=0x4079 runnable [0x00007f82183c2000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051cfa6f88> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051cfa6f70> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051ba67370> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-14" prio=10 tid=0x00007f8588177800 nid=0x4078 runnable [0x00007f82184c3000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051ce5cdd0> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051ce5cdb8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051c6c7bd0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-12" prio=10 tid=0x00007f8584142000 nid=0x4077 runnable [0x00007f82185c4000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051cc27cb8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051cc27ca0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051babd0b8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-18" prio=10 tid=0x00007f85801e6000 nid=0x4076 runnable [0x00007f82186c5000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051cb895c8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051cc77fd0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051c6c7cf0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-11" prio=10 tid=0x00007f8584127800 nid=0x4075 runnable [0x00007f82187c6000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051ca540d0> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051caad368> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051c6c7e10> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-13" prio=10 tid=0x00007f858815c800 nid=0x4074 runnable [0x00007f82188c7000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051c953ab8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051ca799a8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051babd028> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-17" prio=10 tid=0x00007f85801cb800 nid=0x4073 runnable [0x00007f82189c8000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051c77c3d8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051c8137c8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051c3856f0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-10" prio=10 tid=0x00007f858410c800 nid=0x4072 runnable [0x00007f8218ac9000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051c7dd2b0> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051c613988> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051babcde8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-16" prio=10 tid=0x00007f85801b1000 nid=0x4071 runnable [0x00007f8218bca000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051c776580> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051c6633c8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051c6c7ea0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-9" prio=10 tid=0x00007f85840f1800 nid=0x4070 runnable [0x00007f8218ccb000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051c36b550> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051c36b568> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051ba675f8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-8" prio=10 tid=0x00007f85840d6800 nid=0x406f runnable [0x00007f8218dcc000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051c546b00> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051c40a178> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051ba67760> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-7" prio=10 tid=0x00007f85840bb800 nid=0x406e runnable [0x00007f8218ecd000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051c2ecf90> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051c29a100> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051ba675b0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-15" prio=10 tid=0x00007f8580196000 nid=0x406d runnable [0x00007f8218fce000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051c2ee670> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051c2ee688> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051ba67718> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-12" prio=10 tid=0x00007f8588141800 nid=0x406c runnable [0x00007f82190cf000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051c0bd978> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051c16cc00> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051babcf08> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-6" prio=10 tid=0x00007f85840a0800 nid=0x406b runnable [0x00007f82191d0000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051c068508> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051bf825f0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051ba676d0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-11" prio=10 tid=0x00007f8588127000 nid=0x406a runnable [0x00007f82192d1000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051becaa68> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051c03ead0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051babce78> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-10" prio=10 tid=0x00007f858810c800 nid=0x4069 runnable [0x00007f82193d2000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051be31b98> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051bd56790> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051ba67490> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-14" prio=10 tid=0x00007f858017b000 nid=0x4068 runnable [0x00007f82194d3000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051bccb8c0> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051bb14420> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051babcd58> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-5" prio=10 tid=0x00007f8584085800 nid=0x4067 runnable [0x00007f82195d4000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051bd004a8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051bd49480> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051ba67400> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-4" prio=10 tid=0x00007f858406b000 nid=0x4066 runnable [0x00007f82196d5000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051ba1e928> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051bccdd00> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051ba66ef0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-9" prio=10 tid=0x00007f85880f1800 nid=0x4065 waiting for monitor entry [0x00007f82197d5000] 
> > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
> > > - waiting to lock <0x0000000519111590> (a org.apache.flume.channel.file.LogFileV3$Writer)
> > > at org.apache.flume.channel.file.Log.put(Log.java:507)
> > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
> > > at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> > > at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> > > at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> > > at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> > > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> > > at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> > > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> > > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> > > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-8" prio=10 tid=0x00007f85880d6800 nid=0x4064 runnable [0x00007f82198d7000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051b5bcca8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051b7ef8f0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051b5c7cb8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-7" prio=10 tid=0x00007f85880bb800 nid=0x4063 runnable [0x00007f82199d8000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051b788ce8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051b788d00> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051b285260> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-13" prio=10 tid=0x00007f8580160800 nid=0x4062 runnable [0x00007f8219ad9000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051b71a0f8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051b750e98> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051aed5a70> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-10" prio=10 tid=0x00007f85d810b800 nid=0x4061 runnable [0x00007f8219bda000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051b517878> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051b5bab30> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051aed5b90> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-9" prio=10 tid=0x00007f85d80f0800 nid=0x4060 runnable [0x00007f8219cdb000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051b5656e0> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051b5657a0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051b285338> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-8" prio=10 tid=0x00007f85d80d5800 nid=0x405f runnable [0x00007f8219ddc000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051b3df220> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051b475118> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051aed5c20> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-6" prio=10 tid=0x00007f85880a0800 nid=0x405e runnable [0x00007f8219edd000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051b11c010> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051b1eab78> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051af91478> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-7" prio=10 tid=0x00007f85d80bb000 nid=0x405d runnable [0x00007f8219fde000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051b1d36e0> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051b0328c0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051aed5cb0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-12" prio=10 tid=0x00007f8580145800 nid=0x405c runnable [0x00007f821a0df000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051aeb3070> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051aeb3088> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051a98a820> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-6" prio=10 tid=0x00007f85d80a0000 nid=0x405b runnable [0x00007f821a1e0000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051aefa920> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051af3ee40> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051ac28810> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-5" prio=10 tid=0x00007f8588086000 nid=0x405a runnable [0x00007f821a2e1000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051aefe1b0> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051ad2b858> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051aed5e18> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-11" prio=10 tid=0x00007f858012a800 nid=0x4059 runnable [0x00007f821a3e2000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051ac42588> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051ad05d70> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051a98a7d8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-3" prio=10 tid=0x00007f8584050000 nid=0x4058 runnable [0x00007f821a4e3000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051ad06100> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051ad47428> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051ac289c0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-10" prio=10 tid=0x00007f858010f800 nid=0x4057 runnable [0x00007f821a5e4000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051aab5d80> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051ab8ce38> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051a98a748> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-5" prio=10 tid=0x00007f85d8085800 nid=0x4056 runnable [0x00007f821a6e5000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051a962980> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051a775fb0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051a842098> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-2" prio=10 tid=0x00007f8584035000 nid=0x4055 runnable [0x00007f821a7e6000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051a817210> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051a7d2b48> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051a841ca8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-4" prio=10 tid=0x00007f85d806b000 nid=0x4054 runnable [0x00007f821a8e7000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051a903340> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051a903358> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051a42db50> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-9" prio=10 tid=0x00007f85800f4800 nid=0x4053 runnable [0x00007f821a9e8000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051a4f17d8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051a37ae08> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051a22e340> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-4" prio=10 tid=0x00007f858806b000 nid=0x4052 runnable [0x00007f821aae9000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051a4d81e8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051a616bb0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051a1dfb40> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-3" prio=10 tid=0x00007f85d8050000 nid=0x4051 runnable [0x00007f821abea000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051a4fb240> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051a3913b8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051a213878> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-2" prio=10 tid=0x00007f85d8035800 nid=0x4050 runnable [0x00007f821aceb000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051a506248> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051a506308> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051a42dd48> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-8" prio=10 tid=0x00007f85800d9800 nid=0x404f runnable [0x00007f821adec000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051a2d5170> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051a2d5188> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051a22e2b0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-7" prio=10 tid=0x00007f85800be800 nid=0x404e runnable [0x00007f821aeed000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051a2d3cb8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051a2b8310> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051a213758> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10 tid=0x00007f857b378800 nid=0x404d waiting for monitor entry [0x00007f821afee000] 
> > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > at org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
> > > - waiting to lock <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
> > > at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> > > at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> > > at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
> > > at org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> > > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f836c01a000 nid=0x404c runnable [0x00007f821b0ef000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051a045bb0> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051a045bc8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051a000c98> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b376800 nid=0x404b waiting on condition [0x00007f821b1f0000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051a045c10> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10 tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.NativeThread.current(Native Method)
> > > at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
> > > at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
> > > - locked <0x00000005190ec998> (a java.lang.Object)
> > > at org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
> > > at org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
> > > - locked <0x0000000519111590> (a org.apache.flume.channel.file.LogFileV3$Writer)
> > > at org.apache.flume.channel.file.Log.take(Log.java:550)
> > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
> > > at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> > > at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> > > at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
> > > at org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> > > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8354035000 nid=0x4049 runnable [0x00007f821b3f2000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051a045c28> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051a045c40> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051a000ce0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834801a000 nid=0x4048 runnable [0x00007f821b4f3000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051a17bd60> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051a17bd78> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051a180230> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b374800 nid=0x4047 waiting on condition [0x00007f821b5f4000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051a08ba88> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857338a800 nid=0x4046 waiting on condition [0x00007f821b6f5000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051a147940> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10 tid=0x00007f85f3bff800 nid=0x4045 waiting on condition [0x00007f821b7f6000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000056dca65c0> (a java.util.concurrent.FutureTask$Sync)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
> > > at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
> > > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:227)
> > > at java.util.concurrent.FutureTask.get(FutureTask.java:91)
> > > at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:323)
> > > at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:269)
> > > at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:352)
> > > at org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> > > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8344035000 nid=0x4044 runnable [0x00007f821b8f7000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051a184a90> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051a184aa8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051a0bb2e0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bfd800 nid=0x4043 waiting on condition [0x00007f821b9f8000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051a147958> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f833001a000 nid=0x4042 runnable [0x00007f821baf9000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051a188f28> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051a188f40> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519d90758> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8334035000 nid=0x4041 runnable [0x00007f821bbfa000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051a045cd0> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051a045ce8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051a000d28> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b372800 nid=0x4040 waiting on condition [0x00007f821bcfb000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519f74590> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573388800 nid=0x403f waiting on condition [0x00007f821bdfc000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519fb0c90> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832c035000 nid=0x403e runnable [0x00007f821befd000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051a045d30> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051a045d48> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051a000d70> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bfb800 nid=0x403d waiting on condition [0x00007f821bffe000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519e5d388> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f831801a000 nid=0x403c runnable [0x00007f82341c0000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519ebdb70> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519ebdb88> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519d3a778> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f831001a000 nid=0x403b runnable [0x00007f82342c1000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519f47f90> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519f47fa8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519e11ca8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaf0000 nid=0x403a waiting on condition [0x00007f82343c2000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519f73018> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8308035000 nid=0x4039 runnable [0x00007f82344c3000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519f745a8> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519f745c0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519ca9f78> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573386800 nid=0x4038 waiting on condition [0x00007f82345c4000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519fa5ce0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bf9800 nid=0x4037 waiting on condition [0x00007f82346c5000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519fb0ca8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8300035000 nid=0x4036 runnable [0x00007f82347c6000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519f22d20> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519f22d38> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519d9f4b8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaee000 nid=0x4035 waiting on condition [0x00007f82348c7000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519fb0cc0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82fc01a000 nid=0x4034 runnable [0x00007f82349c8000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519f74608> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519f74620> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519ca9fc0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82f001a000 nid=0x4033 runnable [0x00007f8234ac9000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519fb0cd8> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519fb0cf0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519cadd40> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f44800 nid=0x4032 waiting on condition [0x00007f8234bca000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519ebdbd0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3377000 nid=0x4031 waiting on condition [0x00007f8234ccb000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519f701c0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82e8035000 nid=0x4030 runnable [0x00007f8234dcc000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519f47ff0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519f48008> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519e11cf0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaec000 nid=0x402f waiting on condition [0x00007f8234ecd000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519fa5cf8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10 tid=0x00007f857fc38000 nid=0x402e waiting for monitor entry [0x00007f8234fce000] 
> > >    java.lang.Thread.State: BLOCKED (on object monitor)
> > > at org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
> > > - waiting to lock <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> > > at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
> > > at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> > > at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> > > at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
> > > at org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> > > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82dc01a000 nid=0x402d runnable [0x00007f82350cf000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519ebdbe8> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519ebdc00> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519d3a7c0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82d001a000 nid=0x402c runnable [0x00007f82351d0000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519fb0d38> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519fb0d50> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519cadd88> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-144-thread-1" prio=10 tid=0x00007f85d8001000 nid=0x402b runnable [0x00007f82352d1000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519f481a0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519f49040> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519e11d80> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6d3800 nid=0x402a waiting on condition [0x00007f82353d2000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519f22d80> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f42800 nid=0x4029 waiting on condition [0x00007f82354d3000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519fa5d10> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82cc035000 nid=0x4028 runnable [0x00007f82355d4000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519fb0d98> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519fb0db0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519caddd0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-3" prio=10 tid=0x00007f8588001000 nid=0x4027 runnable [0x00007f82356d5000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519ebdc48> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519ebdc60> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519d3a808> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3375000 nid=0x4026 waiting on condition [0x00007f82357d6000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519fa5d28> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82c4035000 nid=0x4025 runnable [0x00007f82358d7000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519f22d98> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519f22db0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519d9f500> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaea000 nid=0x4024 waiting on condition [0x00007f82359d8000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519fa5d40> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a8035000 nid=0x4023 runnable [0x00007f8235ad9000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519fb0df8> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519fb0e10> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519cade18> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82b401a000 nid=0x4022 runnable [0x00007f8235bda000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519f73030> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519f73048> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519e69de8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82ac035000 nid=0x4021 runnable [0x00007f8235cdb000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519f61380> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519f61398> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005197600f0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6d1800 nid=0x4020 waiting on condition [0x00007f8235ddc000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519f22df8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f40800 nid=0x401f waiting on condition [0x00007f8235edd000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519fa5d58> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3373000 nid=0x401e waiting on condition [0x00007f8235fde000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519ebdca8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a4035000 nid=0x401d runnable [0x00007f82360df000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519fb0e58> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519fb0e70> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519cade60> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a8000 nid=0x401c waiting on condition [0x00007f82361e0000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519f22e10> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-6" prio=10 tid=0x00007f85800a3800 nid=0x401b runnable [0x00007f82362e1000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519ebde28> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519de7350> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519d3a898> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828801a000 nid=0x401a runnable [0x00007f82363e2000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519fa5d70> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519fa5d88> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519d36338> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828c035000 nid=0x4019 runnable [0x00007f82364e3000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519f73090> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519f730a8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519e69e30> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8280035000 nid=0x4018 runnable [0x00007f82365e4000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519fb0eb8> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519e1a7f8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519cadea8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6cf800 nid=0x4017 waiting on condition [0x00007f82366e5000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519f22e28> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f3e800 nid=0x4016 waiting on condition [0x00007f82367e6000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519f701d8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3371000 nid=0x4015 waiting on condition [0x00007f82368e7000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519fb0f00> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8278035000 nid=0x4014 runnable [0x00007f82369e8000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519e93628> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519e93640> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519d9f548> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a6000 nid=0x4013 waiting on condition [0x00007f8236ae9000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005199bdaf0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f827001a000 nid=0x4012 runnable [0x00007f8236bea000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519fb0f18> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519fb0f30> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519cadef0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6cd800 nid=0x4011 waiting on condition [0x00007f8236ceb000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519c47f38> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826c035000 nid=0x4010 runnable [0x00007f8236dec000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519fa5dd0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519fa5de8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519d36380> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826001a000 nid=0x400f runnable [0x00007f8236eed000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519c8c170> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519c8c188> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519a73d78> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572aeb000 nid=0x400e waiting on condition [0x00007f8236fee000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519c7e570> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2e800 nid=0x400d waiting on condition [0x00007f82370ef000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519c47f50> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8258035000 nid=0x400b runnable [0x00007f82371f0000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519c4b190> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519c4b1a8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005199eb068> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a4000 nid=0x400a waiting on condition [0x00007f82372f1000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519c47f68> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f825001a000 nid=0x4009 runnable [0x00007f82373f2000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519c33550> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519c33568> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051996b7b8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e29c000 nid=0x4008 waiting on condition [0x00007f82374f3000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519c47f80> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f824c03f800 nid=0x4007 runnable [0x00007f82375f4000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519c7e588> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519c7e5a0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005199a1910> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8240035000 nid=0x4006 runnable [0x00007f82376f5000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519c47f98> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519c47fb0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005199536b8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae9000 nid=0x4005 waiting on condition [0x00007f82377f6000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519c368c8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2c800 nid=0x4004 waiting on condition [0x00007f82378f7000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519c29bf0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8238035000 nid=0x4003 runnable [0x00007f82379f8000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519bd8198> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519bd81b0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519b59da0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a2000 nid=0x4002 waiting on condition [0x00007f8237af9000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519c7e5e8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8230035000 nid=0x4001 runnable [0x00007f8237bfa000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519c47ff8> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519c48010> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519953700> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e29a000 nid=0x4000 waiting on condition [0x00007f8237cfb000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519c7e600> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f823001a000 nid=0x3fff runnable [0x00007f8237dfc000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519c7e618> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519c7e630> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005199a1958> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f823c01a000 nid=0x3ffe runnable [0x00007f8237efd000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519c48058> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519c48070> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519a6d798> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f823801a000 nid=0x3ffd runnable [0x00007f8237ffe000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519c335b0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519c335c8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051996b800> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae7000 nid=0x3ffc waiting on condition [0x00007f84001c0000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519c480b8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2a800 nid=0x3ffb waiting on condition [0x00007f84002c1000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519bd81f8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24e800 nid=0x3ffa waiting on condition [0x00007f84003c2000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519c29c08> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f824001a000 nid=0x3ff9 runnable [0x00007f84004c3000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519c7e678> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519c7e690> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005199a19a0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e298000 nid=0x3ff8 waiting on condition [0x00007f84005c4000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519c67408> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-5" prio=10 tid=0x00007f8580088800 nid=0x3ff7 runnable [0x00007f84006c5000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519c48238> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519d5e0d0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519a6d828> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f825c01a000 nid=0x3ff6 runnable [0x00007f84007c6000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519c368e0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519c368f8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519953088> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f825801a000 nid=0x3ff5 runnable [0x00007f84008c7000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519c33610> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519c33628> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051996b848> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826401a000 nid=0x3ff4 runnable [0x00007f84009c8000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519c7e6d8> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519c7e6f0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005199a19e8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae5000 nid=0x3ff3 waiting on condition [0x00007f8400ac9000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519c730b8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24c800 nid=0x3ff2 waiting on condition [0x00007f8400bca000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519c29c20> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f29000 nid=0x3ff1 waiting on condition [0x00007f8400ccb000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519c4b1f0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826c01a000 nid=0x3ff0 runnable [0x00007f8400dcc000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519bd8210> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519bd8228> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519b59de8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e296800 nid=0x3fef waiting on condition [0x00007f8400ecd000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519c8c1d0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f827801a000 nid=0x3fee runnable [0x00007f8400fce000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519c7e738> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519c7e750> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005199a1a30> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828401a000 nid=0x3fed runnable [0x00007f84010cf000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519c33670> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519c33688> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051996b890> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828001a000 nid=0x3fec runnable [0x00007f84011d0000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519c29c38> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519c29c50> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519955c68> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828c01a000 nid=0x3feb runnable [0x00007f84012d1000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519c7e798> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519c7e7b0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005199533a0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae6000 nid=0x3fea waiting on condition [0x00007f84013d2000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519bd8270> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24a800 nid=0x3fe9 waiting on condition [0x00007f84014d3000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519c36940> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572273800 nid=0x3fe8 waiting on condition [0x00007f84015d4000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519c8c1e8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de54800 nid=0x3fe7 waiting on condition [0x00007f84016d5000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519c67420> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f829001a000 nid=0x3fe6 runnable [0x00007f84017d6000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519c7e7f8> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519c7e810> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005199533e8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a401a000 nid=0x3fe5 runnable [0x00007f84018d7000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519c336d0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519c336e8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051996b8d8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae4000 nid=0x3fe4 waiting on condition [0x00007f84019d8000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519946350> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-4" prio=10 tid=0x00007f858006d800 nid=0x3fe3 runnable [0x00007f8401ad9000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519bd83d8> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519f042a0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519b59e78> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82ac01a000 nid=0x3fe2 runnable [0x00007f8401bda000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519c7e858> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519c7e870> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519953430> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a248800 nid=0x3fe1 waiting on condition [0x00007f8401cdb000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519890ec8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a801a000 nid=0x3fe0 runnable [0x00007f8401ddc000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519c8c200> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519c8c218> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519a73dc0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572271800 nid=0x3fdf waiting on condition [0x00007f8401edd000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005198a3170> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de52800 nid=0x3fde waiting on condition [0x00007f8401fde000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005198b7610> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82b801a000 nid=0x3fdd runnable [0x00007f84020df000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519947f70> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519947f88> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005197bc938> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82c401a000 nid=0x3fdc runnable [0x00007f84021e0000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519946368> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519946380> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051983b9b8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82cc01a000 nid=0x3fdb runnable [0x00007f84022e1000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005198a3188> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005198a31a0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005197300e0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae2000 nid=0x3fda waiting on condition [0x00007f84023e2000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005199463c8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857226f800 nid=0x3fd9 waiting on condition [0x00007f84024e3000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519792380> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82c801a000 nid=0x3fd8 runnable [0x00007f84025e4000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519947fd0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519947fe8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005197bc980> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579597000 nid=0x3fd7 waiting on condition [0x00007f84026e5000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519488cc0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de50800 nid=0x3fd6 waiting on condition [0x00007f84027e6000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005198cc640> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82d801a000 nid=0x3fd5 runnable [0x00007f84028e7000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005199463e0> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005199463f8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051983ba00> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82e401a000 nid=0x3fd4 runnable [0x00007f84029e8000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005198f7fc8> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005198f7fe0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005194d0160> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82ec01a000 nid=0x3fd3 runnable [0x00007f8402ae9000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005198a31e8> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005198a3200> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519730128> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae0800 nid=0x3fd2 waiting on condition [0x00007f8402bea000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519948030> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857226d800 nid=0x3fd1 waiting on condition [0x00007f8402ceb000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519946440> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82e801a000 nid=0x3fd0 runnable [0x00007f8402dec000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519890ee0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519890ef8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005196f56f8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579595000 nid=0x3fcf waiting on condition [0x00007f8402eed000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005198b7628> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de4e800 nid=0x3fce waiting on condition [0x00007f8402fee000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005198a3248> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830401a000 nid=0x3fcd runnable [0x00007f84030ef000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519946458> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519946470> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051983ba48> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830001a000 nid=0x3fcc runnable [0x00007f84031f0000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519937b88> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519937ba0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519887cb8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830c01a000 nid=0x3fcb runnable [0x00007f84032f1000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519948048> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519948060> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005197bc9c8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830801a000 nid=0x3fca runnable [0x00007f84033f2000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005199464b8> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005199464d0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051983ba90> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e2b000 nid=0x3fc9 waiting on condition [0x00007f84034f3000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519943980> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1595000 nid=0x3fc8 waiting on condition [0x00007f84035f4000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005198a3260> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857bff5800 nid=0x3fc7 waiting on condition [0x00007f84036f5000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519901138> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da0c000 nid=0x3fc6 waiting on condition [0x00007f84037f6000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005199480a8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832001a000 nid=0x3fc5 runnable [0x00007f84038f7000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519946518> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519946530> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051983bad8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832c01a000 nid=0x3fc4 runnable [0x00007f84039f8000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005198a3278> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005198a3290> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519730170> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832801a000 nid=0x3fc3 runnable [0x00007f8403af9000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005198f8028> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005198f8040> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005194d01a8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f833401a000 nid=0x3fc2 runnable [0x00007f8403bfa000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005198b7640> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005198b7658> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005196f3480> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e29000 nid=0x3fc1 waiting on condition [0x00007f8403cfb000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519946578> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1593000 nid=0x3fc0 waiting on condition [0x00007f8403dfc000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005199480c0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579592000 nid=0x3fbf waiting on condition [0x00007f8403efd000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519943998> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da0a800 nid=0x3fbe waiting on condition [0x00007f84401c0000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519946590> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834401a000 nid=0x3fbd runnable [0x00007f84402c1000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519903410> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519903428> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005191a1498> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834001a000 nid=0x3fbc runnable [0x00007f84403c2000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519943ef0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519943f08> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005196f5740> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834c01a000 nid=0x3fbb runnable [0x00007f84404c3000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005199497b0> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005199497c8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519685f60> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573ff1800 nid=0x3fba waiting on condition [0x00007f84405c4000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005198f8088> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1591800 nid=0x3fb9 waiting on condition [0x00007f84406c5000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005198a32d8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579591000 nid=0x3fb8 waiting on condition [0x00007f84407c6000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005199480d8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f835401a000 nid=0x3fb7 runnable [0x00007f84408c7000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005199465a8> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005199465c0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051983bb20> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da08800 nid=0x3fb6 waiting on condition [0x00007f84409c8000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005199439b0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-142-thread-1" prio=10 tid=0x00007f8584001000 nid=0x3fb5 runnable [0x00007f8440ac9000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005198b77f0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519a00960> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519952e48> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f837401a000 nid=0x3fb4 runnable [0x00007f8440bca000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519937be8> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519937c00> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519887d00> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f836801a000 nid=0x3fb3 runnable [0x00007f8440ccb000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519943f50> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519943f68> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005196f5788> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f837001a000 nid=0x3fb2 runnable [0x00007f8440dcc000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519901150> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519901168> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519106100> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f158f800 nid=0x3fb1 waiting on condition [0x00007f8440ecd000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519946608> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e26800 nid=0x3fb0 waiting on condition [0x00007f8440fce000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005198a32f0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913e000 nid=0x3faf waiting on condition [0x00007f84410cf000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005199439c8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f837801a000 nid=0x3fae runnable [0x00007f84411d0000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005199480f0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519948108> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005197bca10> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da07000 nid=0x3fad waiting on condition [0x00007f84412d1000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519943fb0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f838801a000 nid=0x3fac runnable [0x00007f84413d2000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005198f80a0> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005198f80b8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005194d01f0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f839401a000 nid=0x3fab runnable [0x00007f84414d3000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519946620> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519946638> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051983bb68> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f839001a000 nid=0x3faa runnable [0x00007f84415d4000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519937c48> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519937c60> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519887d48> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d0d000 nid=0x3fa9 waiting on condition [0x00007f84416d5000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519654310> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913c000 nid=0x3fa8 waiting on condition [0x00007f84417d6000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005191b54d8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571164000 nid=0x3fa7 waiting on condition [0x00007f84418d7000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519637a20> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83a801a000 nid=0x3fa6 runnable [0x00007f84419d8000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005195f76c8> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005196430c0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005194ecaf0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83b401a000 nid=0x3fa5 runnable [0x00007f8441ad9000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051965f960> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051965f978> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519104648> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83b001a000 nid=0x3fa4 runnable [0x00007f8441bda000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519655e28> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519655e40> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519104ba0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-3" prio=10 tid=0x00007f8580052800 nid=0x3fa3 runnable [0x00007f8441cdb000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519637b10> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519694210> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051916e518> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83bc01a000 nid=0x3fa2 runnable [0x00007f8441ddc000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519597678> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519597690> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519106388> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c4800 nid=0x3fa1 waiting on condition [0x00007f8441edd000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519654328> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d0b000 nid=0x3fa0 waiting on condition [0x00007f8441fde000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519676ad0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571162000 nid=0x3f9f waiting on condition [0x00007f84420df000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519655618> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913a000 nid=0x3f9e waiting on condition [0x00007f84421e0000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005194ae248> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-2" prio=10 tid=0x00007f8580001000 nid=0x3f9d runnable [0x00007f84422e1000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519654340> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519654358> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519105380> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83cc01a000 nid=0x3f9c runnable [0x00007f84423e2000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519655e88> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519655ea0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519104be8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83d001a000 nid=0x3f9b runnable [0x00007f84424e3000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005194e1350> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005194e1368> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005191041c8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c3000 nid=0x3f9a waiting on condition [0x00007f84425e4000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051967caf8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83d801a000 nid=0x3f99 runnable [0x00007f84426e5000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519655ee8> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519655f00> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519104c30> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83dc01a000 nid=0x3f98 runnable [0x00007f84427e6000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005196543a0> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005196543b8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005191053c8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d09800 nid=0x3f97 waiting on condition [0x00007f84428e7000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519655630> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579138000 nid=0x3f96 waiting on condition [0x00007f84429e8000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005196592c8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571160000 nid=0x3f95 waiting on condition [0x00007f8442ae9000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519655f48> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f840801a000 nid=0x3f94 runnable [0x00007f8442bea000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519655648> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519655660> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051914b290> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83e801a000 nid=0x3f93 runnable [0x00007f8442ceb000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519643108> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519643120> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005194ecb38> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d07800 nid=0x3f92 waiting on condition [0x00007f8442dec000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519658428> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83f801a000 nid=0x3f91 runnable [0x00007f8442eed000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005194c9908> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005194c9920> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519511968> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83fc01a000 nid=0x3f90 runnable [0x00007f8442fee000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519643168> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519643180> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005194ecb80> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578465000 nid=0x3f8f waiting on condition [0x00007f8403ffe000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519654400> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857115e800 nid=0x3f8e waiting on condition [0x00007f84430ef000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005196556a8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c1000 nid=0x3f8d waiting on condition [0x00007f84431f0000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519654418> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f840c01a000 nid=0x3f8c runnable [0x00007f84432f1000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005196431c8> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005196431e0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005194ecbc8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f841801a000 nid=0x3f8b runnable [0x00007f84433f2000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519655f60> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519655f78> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519104c78> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b4000 nid=0x3f8a waiting on condition [0x00007f84434f3000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005196721e8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5bf000 nid=0x3f89 waiting on condition [0x00007f84435f4000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519654430> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f841c01a000 nid=0x3f88 runnable [0x00007f84436f5000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005196556c0> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005196556d8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051914b2d8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f842801a000 nid=0x3f87 runnable [0x00007f84437f6000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519643228> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519643240> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005194ecc10> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b2000 nid=0x3f86 waiting on condition [0x00007f84438f7000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519654448> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5bd800 nid=0x3f85 waiting on condition [0x00007f84439f8000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051967cb10> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f843001a000 nid=0x3f84 runnable [0x00007f8443af9000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519655fc0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519655fd8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519104cc0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b0800 nid=0x3f83 waiting on condition [0x00007f8443bfa000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005196592e0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f843801a000 nid=0x3f82 runnable [0x00007f8443cfb000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519643288> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005196432a0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005194ecc58> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d17b000 nid=0x3f81 waiting on condition [0x00007f8443dfc000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051921afd8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f843c01a000 nid=0x3f80 runnable [0x00007f8443efd000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519654460> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519654478> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519105410> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08ae800 nid=0x3f7f waiting on condition [0x00007f8443ffe000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519676ae8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f844401a000 nid=0x3f7e runnable [0x00007f85441a6000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051967cb28> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051967cb40> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005194745e0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d179000 nid=0x3f7d waiting on condition [0x00007f85442a7000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005196432e8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f845801a000 nid=0x3f7c runnable [0x00007f85443a8000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519658440> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005196583f8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519349128> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f845401a000 nid=0x3f7b runnable [0x00007f85444a9000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519676aa0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519676ab8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051921b020> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08ad800 nid=0x3f7a waiting on condition [0x00007f85446c5000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519659298> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578463000 nid=0x3f79 waiting on condition [0x00007f85447c6000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005194a3568> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f845c01a000 nid=0x3f78 runnable [0x00007f85445aa000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005196544c0> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005196544d8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519105458> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d177000 nid=0x3f77 waiting on condition [0x00007f85448c7000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519656020> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f846401a000 nid=0x3f76 runnable [0x00007f85449c8000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519655720> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519655738> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519105f98> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857115c800 nid=0x3f75 waiting on condition [0x00007f8544ac9000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519402f58> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f847801a000 nid=0x3f74 runnable [0x00007f8544bca000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519654520> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519654538> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005191054a0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f847401a000 nid=0x3f73 runnable [0x00007f8544ccb000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at java.util.HashMap.newKeyIterator(HashMap.java:840)
> > > at java.util.HashMap$KeySet.iterator(HashMap.java:874)
> > > at java.util.HashSet.iterator(HashSet.java:153)
> > > at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
> > > - locked <0x0000000519656248> (a java.util.HashSet)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519656038> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519656050> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519104d08> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0015800 nid=0x3f72 waiting on condition [0x00007f8544dcc000]
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051941d010> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578461000 nid=0x3f71 waiting on condition [0x00007f8544ecd000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519477938> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f847c01a000 nid=0x3f70 runnable [0x00007f8544fce000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051942aa08> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051942aa20> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519349050> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d175800 nid=0x3f6f waiting on condition [0x00007f85450cf000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519470c60> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f848401a000 nid=0x3f6e runnable [0x00007f85451d0000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519445fa0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519445fb8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005194679d0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570456800 nid=0x3f6d waiting on condition [0x00007f85452d1000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519419230> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f849801a000 nid=0x3f6c runnable [0x00007f85453d2000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519470c78> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519470c90> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005194743a0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f849401a000 nid=0x3f6b runnable [0x00007f85454d3000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051945beb8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051945bed0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519196000> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0013800 nid=0x3f6a waiting on condition [0x00007f85455d4000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519470cd8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857845f000 nid=0x3f69 waiting on condition [0x00007f85456d5000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051941d1c0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f849c01a000 nid=0x3f68 runnable [0x00007f85457d6000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005194583b8> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005194583d0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005194743e8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c488000 nid=0x3f67 waiting on condition [0x00007f85458d7000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519424e80> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84a401a000 nid=0x3f66 runnable [0x00007f85459d8000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051941d1d8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051941d1f0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519467a18> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570455000 nid=0x3f65 waiting on condition [0x00007f8545ad9000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051945bf18> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84b801a000 nid=0x3f64 runnable [0x00007f8545bda000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519419248> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519419200> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519431ee0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84c001a000 nid=0x3f63 runnable [0x00007f8545cdb000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051941d238> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051941d250> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519467a60> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0012000 nid=0x3f62 waiting on condition [0x00007f8545ddc000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519458418> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84bc01a000 nid=0x3f61 runnable [0x00007f8545edd000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051945bf30> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051945bf48> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519196048> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857845e000 nid=0x3f60 waiting on condition [0x00007f8545fde000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519479f10> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c486000 nid=0x3f5f waiting on condition [0x00007f85460df000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051941d298> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84c401a000 nid=0x3f5e runnable [0x00007f85461e0000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051942a9a8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051942a9c0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519349098> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570453800 nid=0x3f5d waiting on condition [0x00007f85462e1000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051945bf90> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84cc01a000 nid=0x3f5c runnable [0x00007f85463e2000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519458430> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519458448> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519474430> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c484800 nid=0x3f5b waiting on condition [0x00007f85464e3000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051941d2b0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84d401a000 nid=0x3f5a runnable [0x00007f85465e4000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005194191e8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051936a4e0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519431f28> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570450800 nid=0x3f59 waiting on condition [0x00007f85466e5000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051943ef60> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84dc01a000 nid=0x3f58 runnable [0x00007f85467e6000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051942a948> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051942a960> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005193490e0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c482800 nid=0x3f57 waiting on condition [0x00007f85468e7000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051945bfa8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84e401a000 nid=0x3f56 runnable [0x00007f85469e8000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051941d2c8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051941d2e0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519467aa8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857044f000 nid=0x3f55 waiting on condition [0x00007f8546ae9000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051945bfc0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84f801a000 nid=0x3f54 runnable [0x00007f8546bea000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051936a4c8> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051936a480> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519431f70> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84f401a000 nid=0x3f53 runnable [0x00007f8546ceb000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519458490> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005194584a8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519474478> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578008000 nid=0x3f52 waiting on condition [0x00007f8546dec000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051942a8e8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c00d000 nid=0x3f51 waiting on condition [0x00007f8546eed000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519447098> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84fc01a000 nid=0x3f50 runnable [0x00007f8546fee000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051941d328> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051941d340> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519467af0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857044d000 nid=0x3f4f waiting on condition [0x00007f85470ef000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051945bfd8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f851001a000 nid=0x3f4e runnable [0x00007f85471f0000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519424e98> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519424e50> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519380d20> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f850c01a000 nid=0x3f4d runnable [0x00007f85472f1000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005194584f0> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519458508> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005194744c0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578006800 nid=0x3f4c waiting on condition [0x00007f85473f2000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051945bff0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c01d800 nid=0x3f4b waiting on condition [0x00007f85474f3000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051936a468> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f851401a000 nid=0x3f4a runnable [0x00007f85475f4000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051941d388> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051941d3a0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519467b38> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857045b800 nid=0x3f49 waiting on condition [0x00007f85476f5000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051942a900> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f853801a000 nid=0x3f48 runnable [0x00007f85477f6000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519479f28> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519479ee0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519203068> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f853001f800 nid=0x3f47 runnable [0x00007f85478f7000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519477950> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519477968> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051922faf0> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f852c01f000 nid=0x3f46 runnable [0x00007f85479f8000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005194353f8> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519435410> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x000000051921b0f8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f853401a000 nid=0x3f45 runnable [0x00007f8547af9000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051945c008> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051945c020> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000519196090> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578005800 nid=0x3f44 waiting on condition [0x00007f8547bfa000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005190de788> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f001a800 nid=0x3f43 waiting on condition [0x00007f8547cfb000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000518fae040> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c022000 nid=0x3f42 waiting on condition [0x00007f8547dfc000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000518cafb20> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570460000 nid=0x3f41 waiting on condition [0x00007f8547efd000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005190db9c8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> > > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> > > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-2" prio=10 tid=0x00007f8588036000 nid=0x3f40 runnable [0x00007f8547ffe000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005190ee0e8> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005191f0be8> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005190e1d18> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-140-thread-1" prio=10 tid=0x00007f858801b800 nid=0x3f3f runnable [0x00007f858c18a000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000518f16658> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051941f820> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005190edc80> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-139-thread-1" prio=10 tid=0x00007f8580002800 nid=0x3f3e runnable [0x00007f858c28b000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x000000051913eb50> (a sun.nio.ch.Util$2)
> > > - locked <0x000000051913eca0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000518c9c4e8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> > > at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> > > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-148-thread-1" prio=10 tid=0x00007f856844c000 nid=0x3f3d waiting on condition [0x00007f858c38c000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x00000005190cfe98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-146-thread-1" prio=10 tid=0x00007f85dc006800 nid=0x3f3c waiting on condition [0x00007f858c48d000]
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519018558> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-147-thread-1" prio=10 tid=0x00007f85e82b1000 nid=0x3f3b waiting on condition [0x00007f858c58e000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051903b488> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-145-thread-1" prio=10 tid=0x00007f8574027000 nid=0x3f3a waiting on condition [0x00007f858c68f000]
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051903b998> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-141-thread-1" prio=10 tid=0x00007f85e8159000 nid=0x3f39 runnable [0x00007f861cd8e000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005190db9e0> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005190db998> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000518c9c020> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-138-thread-1" prio=10 tid=0x00007f856844a800 nid=0x3f38 runnable [0x00007f861cf90000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x00000005190e1b68> (a sun.nio.ch.Util$2)
> > > - locked <0x00000005190e1b80> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x0000000518c9c608> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-143-thread-1" prio=10 tid=0x00007f85dc003000 nid=0x3f37 runnable [0x00007f861d091000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519018fa8> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519018fc0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005190e2228> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "pool-137-thread-1" prio=10 tid=0x00007f8574007000 nid=0x3f36 runnable [0x00007f861ce8f000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > > - locked <0x0000000519003dc8> (a sun.nio.ch.Util$2)
> > > - locked <0x0000000519003de0> (a java.util.Collections$UnmodifiableSet)
> > > - locked <0x00000005190ea1a8> (a sun.nio.ch.EPollSelectorImpl)
> > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > > at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
> > > at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "1407844613@qtp-1263282395-1 - Acceptor0 SocketConnector@0.0.0.0:54321 (http://SocketConnector@0.0.0.0:54321)" prio=10 tid=0x00007f85e404a800 nid=0x3f35 runnable [0x00007f858c790000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > at java.net.PlainSocketImpl.socketAccept(Native Method)
> > > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
> > > - locked <0x00000005190e2720> (a java.net.SocksSocketImpl)
> > > at java.net.ServerSocket.implAccept(ServerSocket.java:462)
> > > at java.net.ServerSocket.accept(ServerSocket.java:430)
> > > at org.mortbay.jetty.bio.SocketConnector.accept(SocketConnector.java:99)
> > > at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> > > at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> > > 
> > > "208924888@qtp-1263282395-0" prio=10 tid=0x00007f85e4021800 nid=0x3f34 in Object.wait() [0x00007f861c15b000]
> > >    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> > > at java.lang.Object.wait(Native Method)
> > > - waiting on <0x00000005190d0528> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
> > > at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> > > - locked <0x00000005190d0528> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
> > > 
> > > "lifecycleSupervisor-1-9" prio=10 tid=0x00007f85e401a800 nid=0x3f33 waiting on condition [0x00007f861c687000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "lifecycleSupervisor-1-8" prio=10 tid=0x00007f85e4018800 nid=0x3f32 waiting on condition [0x00007f861c788000]
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "lifecycleSupervisor-1-7" prio=10 tid=0x00007f85e4016800 nid=0x3f31 waiting on condition [0x00007f861c889000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "lifecycleSupervisor-1-6" prio=10 tid=0x00007f85e4014800 nid=0x3f30 waiting on condition [0x00007f861c98a000]
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "lifecycleSupervisor-1-5" prio=10 tid=0x00007f85e4013000 nid=0x3f2f waiting on condition [0x00007f861ca8b000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "lifecycleSupervisor-1-4" prio=10 tid=0x00007f85e4028000 nid=0x3f2e waiting on condition [0x00007f861cb8c000]
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "lifecycleSupervisor-1-3" prio=10 tid=0x00007f85e4026800 nid=0x3f2d waiting on condition [0x00007f861cc8d000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Log-BackgroundWorker-ch1" prio=10 tid=0x00007f85dc001000 nid=0x3f27 waiting on condition [0x00007f861d299000]
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000519017718> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Log-BackgroundWorker-ch2" prio=10 tid=0x00007f85e80ca000 nid=0x3f26 waiting on condition [0x00007f861d39a000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x000000051903b4e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "lifecycleSupervisor-1-2" prio=10 tid=0x00007f85e4025000 nid=0x3f25 waiting on condition [0x00007f861d6ba000]
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "lifecycleSupervisor-1-1" prio=10 tid=0x00007f85e400e800 nid=0x3f24 waiting on condition [0x00007f861d7bb000] 
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "conf-file-poller-0" prio=10 tid=0x00007f85f0001000 nid=0x3f23 waiting on condition [0x00007f861d8bc000]
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000518ec72f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "DestroyJavaVM" prio=10 tid=0x00007f86a800c800 nid=0x3efa waiting on condition [0x0000000000000000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > 
> > > "lifecycleSupervisor-1-0" prio=10 tid=0x00007f86a89f5800 nid=0x3f22 waiting on condition [0x00007f861d9bd000]
> > >    java.lang.Thread.State: TIMED_WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> > > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> > > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > "Low Memory Detector" daemon prio=10 tid=0x00007f86a831d800 nid=0x3f20 runnable [0x0000000000000000]
> > >    java.lang.Thread.State: RUNNABLE
> > > 
> > > "C2 CompilerThread1" daemon prio=10 tid=0x00007f86a831b000 nid=0x3f1f waiting on condition [0x0000000000000000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > 
> > > "C2 CompilerThread0" daemon prio=10 tid=0x00007f86a8319000 nid=0x3f1e waiting on condition [0x0000000000000000]
> > >    java.lang.Thread.State: RUNNABLE
> > > 
> > > "Signal Dispatcher" daemon prio=10 tid=0x00007f86a8316800 nid=0x3f1d runnable [0x0000000000000000]
> > >    java.lang.Thread.State: RUNNABLE
> > > 
> > > "Surrogate Locker Thread (Concurrent GC)" daemon prio=10 tid=0x00007f86a8314800 nid=0x3f1c waiting on condition [0x0000000000000000] 
> > >    java.lang.Thread.State: RUNNABLE
> > > 
> > > "Finalizer" daemon prio=10 tid=0x00007f86a82f9800 nid=0x3f1b in Object.wait() [0x00007f861e3eb000]
> > >    java.lang.Thread.State: WAITING (on object monitor)
> > > at java.lang.Object.wait(Native Method)
> > > - waiting on <0x0000000518c843c0> (a java.lang.ref.ReferenceQueue$Lock)
> > > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> > > - locked <0x0000000518c843c0> (a java.lang.ref.ReferenceQueue$Lock)
> > > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> > > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> > > 
> > > "Reference Handler" daemon prio=10 tid=0x00007f86a82f7800 nid=0x3f1a in Object.wait() [0x00007f861e4ec000]
> > >    java.lang.Thread.State: WAITING (on object monitor)
> > > at java.lang.Object.wait(Native Method)
> > > - waiting on <0x0000000518c80860> (a java.lang.ref.Reference$Lock)
> > > at java.lang.Object.wait(Object.java:485)
> > > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> > > - locked <0x0000000518c80860> (a java.lang.ref.Reference$Lock)
> > > 
> > > "VM Thread" prio=10 tid=0x00007f86a82f1000 nid=0x3f19 runnable 
> > > 
> > > "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f86a801b800 nid=0x3efb runnable  
> > > 
> > > "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f86a801d000 nid=0x3efc runnable 
> > > 
> > > "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f86a801f000 nid=0x3efd runnable  
> > > 
> > > "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f86a8021000 nid=0x3efe runnable 
> > > 
> > > "Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x00007f86a8022800 nid=0x3eff runnable  
> > > 
> > > "Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x00007f86a8024800 nid=0x3f00 runnable 
> > > 
> > > "Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x00007f86a8026000 nid=0x3f01 runnable  
> > > 
> > > "Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x00007f86a8028000 nid=0x3f02 runnable 
> > > 
> > > "Gang worker#8 (Parallel GC Threads)" prio=10 tid=0x00007f86a802a000 nid=0x3f03 runnable  
> > > 
> > > "Gang worker#9 (Parallel GC Threads)" prio=10 tid=0x00007f86a802b800 nid=0x3f04 runnable 
> > > 
> > > "Gang worker#10 (Parallel GC Threads)" prio=10 tid=0x00007f86a802d800 nid=0x3f05 runnable  
> > > 
> > > "Gang worker#11 (Parallel GC Threads)" prio=10 tid=0x00007f86a802f800 nid=0x3f06 runnable 
> > > 
> > > "Gang worker#12 (Parallel GC Threads)" prio=10 tid=0x00007f86a8031000 nid=0x3f07 runnable  
> > > 
> > > "Gang worker#13 (Parallel GC Threads)" prio=10 tid=0x00007f86a8033000 nid=0x3f08 runnable 
> > > 
> > > "Gang worker#14 (Parallel GC Threads)" prio=10 tid=0x00007f86a8035000 nid=0x3f09 runnable  
> > > 
> > > "Gang worker#15 (Parallel GC Threads)" prio=10 tid=0x00007f86a8036800 nid=0x3f0a runnable 
> > > 
> > > "Gang worker#16 (Parallel GC Threads)" prio=10 tid=0x00007f86a8038800 nid=0x3f0b runnable  
> > > 
> > > "Gang worker#17 (Parallel GC Threads)" prio=10 tid=0x00007f86a803a000 nid=0x3f0c runnable 
> > > 
> > > "Gang worker#18 (Parallel GC Threads)" prio=10 tid=0x00007f86a803c000 nid=0x3f0d runnable  
> > > 
> > > "Gang worker#19 (Parallel GC Threads)" prio=10 tid=0x00007f86a803e000 nid=0x3f0e runnable 
> > > 
> > > "Gang worker#20 (Parallel GC Threads)" prio=10 tid=0x00007f86a803f800 nid=0x3f0f runnable  
> > > 
> > > "Gang worker#21 (Parallel GC Threads)" prio=10 tid=0x00007f86a8041800 nid=0x3f10 runnable 
> > > 
> > > "Gang worker#22 (Parallel GC Threads)" prio=10 tid=0x00007f86a8043800 nid=0x3f11 runnable  
> > > 
> > > "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f86a82aa000 nid=0x3f18 runnable 
> > > "Gang worker#0 (Parallel CMS Threads)" prio=10 tid=0x00007f86a829e800 nid=0x3f12 runnable 
> > > 
> > > "Gang worker#1 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a0000 nid=0x3f13 runnable 
> > > 
> > > "Gang worker#2 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a2000 nid=0x3f14 runnable  
> > > 
> > > "Gang worker#3 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a3800 nid=0x3f15 runnable 
> > > 
> > > "Gang worker#4 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a5800 nid=0x3f16 runnable  
> > > 
> > > "Gang worker#5 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a7800 nid=0x3f17 runnable 
> > > 
> > > "VM Periodic Task Thread" prio=10 tid=0x00007f86a8329000 nid=0x3f21 waiting on condition  
> > > 
> > > JNI global references: 1749
> > > 
> > >  
> > > 
> > > -- 
> > > 
> > > P | (415) 677-9222 ext. 205 (tel:%28415%29%20677-9222%20ext.%20205) F | (415) 677-0895 (tel:%28415%29%20677-0895) | pankaj@brightroll.com (mailto:pankaj@brightroll.com) 
> > > Pankaj Gupta | Software Engineer
> > > BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com (http://www.brightroll.com/)
> > > 
> > > United States | Canada | United Kingdom | Germany 
> > > 
> > > We're hiring (http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7)! 
> > 
> 
> 
> 
> -- 
> 
> P | (415) 677-9222 ext. 205 F | (415) 677-0895 | pankaj@brightroll.com (mailto:pankaj@brightroll.com) 
> Pankaj Gupta | Software Engineer
> BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com (http://www.brightroll.com/)
> 
> United States | Canada | United Kingdom | Germany 
> 
> We're hiring (http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7)! 

Re: Lock contention in FileChannel

[Pankaj Gupta <pa...@brightroll.com>]

It seems like some i/o is done inside the lock, which means that time for
taking a lock is proportional to the time for i/o and thus it becomes a
problem. I apologize in advance if I am wrong but the call stack and
behavior I'm seeing seems to suggest that. Specifically, it seems that we
do a write while inside take:
"SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.NativeThread.current(Native Method)
        at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
        at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
        - locked <0x00000005190ec998> (a java.lang.Object)
        at
org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
        at
org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
        - locked <0x0000000519111590> (a
org.apache.flume.channel.file.LogFileV3$Writer)
        at org.apache.flume.channel.file.Log.take(Log.java:550)
        at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
        at
org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
        at
org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
        at
org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
        at
org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
        at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:662)



On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <hshreedharan@cloudera.com
> wrote:

> Since the channel is designed to make sure that events are not duplicated
> to multiple sinks, and to protect against corruption due to concurrency
> issues, we do not need the locking in the channel's flume event queue. It
> is unlikely that locking is what is causing performance issues because the
> channel is heavily I/O bound. If you take a series of thread dumps, you
> will probably see that those threads are moving forward and the ones
> reading/writing from/to disk are the ones which are slower. These locks are
> unlikely to hit performance much.
>
> Thanks,
> Hari
>
> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
>
> Hi,
>
> Spent some more time debugging issues with FileChannel. The problem seems
> to lock contention reading from FlumeEventQueue:
>
> I see a lot of threads like this:
> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
> [0x00007f821afee000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at
> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>         - waiting to lock <0x0000000518ee4c90> (a
> org.apache.flume.channel.file.FlumeEventQueue)
>         at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>         at
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>         at
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>         at
> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>         at
> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:662)
>
>
> I have two file channels and 8 Avro Sinks per file channel. I added more
> sinks because they weren't draining fast enough. It seems like they send
> the batch then wait for an ack before sending again, thus sends are not
> pipelined and having more sinks seemed like a good way of getting some
> parallelism.
>
> Here's the full stack trace:
> 2013-08-13 15:30:32
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02 mixed mode):
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000056d932120> (a
> java.util.concurrent.CountDownLatch$Sync)
>  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>  at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>  at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
>  at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
>  at
> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
> at $Proxy7.appendBatch(Unknown Source)
>  at
> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
> at
> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
>  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000 nid=0x41a1 waiting
> on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>  at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005194c3340> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191a16d8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519174e08> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519174e98> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005195f9670> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519174ce8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191a1918> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191e7f60> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153 runnable
> [0x00007f8210e4d000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
> - locked <0x000000054a2917a8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519174f58> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146 runnable
> [0x00007f8210f4e000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
> - locked <0x0000000546e5dd28> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118 runnable
> [0x00007f821104f000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
> - locked <0x0000000544f763c0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111 runnable
> [0x00007f8211150000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
> - locked <0x000000054473d5a8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110 runnable
> [0x00007f8211251000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
> - locked <0x00000005437f4f70> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f runnable
> [0x00007f8211352000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
> - locked <0x00000005422a9018> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e runnable
> [0x00007f8211453000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
> - locked <0x0000000541063d50> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d runnable
> [0x00007f8211554000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
> - locked <0x0000000540dc1110> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a runnable
> [0x00007f8211655000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
> - locked <0x000000053fbbc3b0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109 runnable
> [0x00007f8211756000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
> - locked <0x000000053f812860> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd runnable
> [0x00007f8211857000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
> - locked <0x000000053e6e5570> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519175138> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519174c28> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191751c8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0 runnable
> [0x00007f8211c5b000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
> - locked <0x0000000536579db0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191750a8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1 runnable
> [0x00007f8211e5d000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
> - locked <0x00000005313943a8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051936f460> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051936dfa8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de runnable
> [0x00007f8212160000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
> - locked <0x000000052dad4468> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd runnable
> [0x00007f8212261000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
> - locked <0x000000052c73c5f0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc runnable
> [0x00007f8212362000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
> - locked <0x000000052b8da998> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db waiting for
> monitor entry [0x00007f8212462000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
> - waiting to lock <0x0000000518ee4c90> (a
> org.apache.flume.channel.file.FlumeEventQueue)
>  at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
> at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>  at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>  at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>  at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051936df18> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
> tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518fe7ae8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8 runnable
> [0x00007f8212766000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
> - locked <0x00000005263d6ba8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7 runnable
> [0x00007f8212867000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
> - locked <0x0000000525d1d0e0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6 runnable
> [0x00007f8212968000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
> - locked <0x0000000525a689d8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5 runnable
> [0x00007f8212a69000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
> - locked <0x0000000524e1e7d8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4 runnable
> [0x00007f8212b6a000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
> - locked <0x0000000524d437a8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-36" prio=10 tid=0x00007f85843ad800 nid=0x40d3 runnable
> [0x00007f8212c6b000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000524cf4140> (a sun.nio.ch.Util$2)
> - locked <0x0000000524cf4128> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000524ab32e8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-35" prio=10 tid=0x00007f85d83a9000 nid=0x40d2 runnable
> [0x00007f8212d6c000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005244877c8> (a sun.nio.ch.Util$2)
> - locked <0x00000005244877b0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000523f5b660> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-34" prio=10 tid=0x00007f85d838e000 nid=0x40d1 runnable
> [0x00007f8212e6d000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052419e490> (a sun.nio.ch.Util$2)
> - locked <0x000000052419e478> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000523f5b6f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-34" prio=10 tid=0x00007f8588390800 nid=0x40d0 waiting for
> monitor entry [0x00007f8212f6d000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
> - waiting to lock <0x0000000519111590> (a
> org.apache.flume.channel.file.LogFileV3$Writer)
>  at org.apache.flume.channel.file.Log.put(Log.java:507)
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
>  at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>  at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>  at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>  at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>  at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-35" prio=10 tid=0x00007f85843ab800 nid=0x40cf runnable
> [0x00007f821306f000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000523d40938> (a sun.nio.ch.Util$2)
> - locked <0x0000000523d40aa0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000523ecec18> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-37" prio=10 tid=0x00007f85803e3800 nid=0x40ce waiting for
> monitor entry [0x00007f821316f000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
> - waiting to lock <0x0000000518ee4c90> (a
> org.apache.flume.channel.file.FlumeEventQueue)
>  at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
> at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>  at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>  at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-34" prio=10 tid=0x00007f8584391000 nid=0x40cd runnable
> [0x00007f8213271000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000523935540> (a sun.nio.ch.Util$2)
>  - locked <0x0000000523935528> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000523788ed8> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-33" prio=10 tid=0x00007f85d8373800 nid=0x40cc runnable
> [0x00007f8213372000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005239ec658> (a sun.nio.ch.Util$2)
>  - locked <0x00000005239ec640> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000523845410> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-33" prio=10 tid=0x00007f8588375800 nid=0x40cb waiting for
> monitor entry [0x00007f8213472000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
>  - waiting to lock <0x0000000519111590> (a
> org.apache.flume.channel.file.LogFileV3$Writer)
> at org.apache.flume.channel.file.Log.put(Log.java:507)
>  at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
> at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>  at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>  at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-33" prio=10 tid=0x00007f8584376000 nid=0x40ca runnable
> [0x00007f8213574000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005237e3028> (a sun.nio.ch.Util$2)
>  - locked <0x00000005237e3010> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000523788e90> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-32" prio=10 tid=0x00007f85d8358800 nid=0x40c9 waiting for
> monitor entry [0x00007f8213674000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>  - waiting to lock <0x0000000518ee4c90> (a
> org.apache.flume.channel.file.FlumeEventQueue)
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>  at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>  at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>  at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>  at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>  at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-32" prio=10 tid=0x00007f858835b000 nid=0x40c8 runnable
> [0x00007f8213776000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000523343688> (a sun.nio.ch.Util$2)
> - locked <0x0000000523343670> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000052305e5c8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-36" prio=10 tid=0x00007f85803c9000 nid=0x40c7 runnable
> [0x00007f8213877000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005230fa580> (a sun.nio.ch.Util$2)
> - locked <0x00000005230fa568> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000522f5ca50> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-32" prio=10 tid=0x00007f858435b000 nid=0x40c6 runnable
> [0x00007f8213978000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005231367a8> (a sun.nio.ch.Util$2)
> - locked <0x0000000523136790> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000052305e658> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-31" prio=10 tid=0x00007f85d833e000 nid=0x40c5 runnable
> [0x00007f8213a79000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000522db30a0> (a sun.nio.ch.Util$2)
> - locked <0x0000000522db3088> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000522bda430> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-31" prio=10 tid=0x00007f8588340000 nid=0x40c4 runnable
> [0x00007f8213b7a000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000522c16518> (a sun.nio.ch.Util$2)
> - locked <0x0000000522c16500> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000522bd97e8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-30" prio=10 tid=0x00007f85d8323800 nid=0x40c3 runnable
> [0x00007f8213c7b000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000522bd1a20> (a sun.nio.ch.Util$2)
> - locked <0x0000000522bd1a08> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000522bda3a0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-31" prio=10 tid=0x00007f8584340800 nid=0x40c2 runnable
> [0x00007f8213d7c000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005228d1608> (a sun.nio.ch.Util$2)
> - locked <0x00000005228d15f0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005228a6aa0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-30" prio=10 tid=0x00007f8588325000 nid=0x40c1 runnable
> [0x00007f8213e7d000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052277b088> (a sun.nio.ch.Util$2)
> - locked <0x000000052277b070> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005225ed3b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-35" prio=10 tid=0x00007f85803ae000 nid=0x40c0 runnable
> [0x00007f8213f7e000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000522737380> (a sun.nio.ch.Util$2)
> - locked <0x0000000522737368> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005225ed328> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-29" prio=10 tid=0x00007f85d8308800 nid=0x40bf runnable
> [0x00007f821407f000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005226034a8> (a sun.nio.ch.Util$2)
> - locked <0x0000000522603490> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005225ed298> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-28" prio=10 tid=0x00007f85d82ed800 nid=0x40be runnable
> [0x00007f8214180000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000522359408> (a sun.nio.ch.Util$2)
> - locked <0x00000005223593f0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005221e7278> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-27" prio=10 tid=0x00007f85d82d2800 nid=0x40bd runnable
> [0x00007f8214281000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005222c0478> (a sun.nio.ch.Util$2)
> - locked <0x00000005222c0460> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005221e6e40> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-29" prio=10 tid=0x00007f858830a800 nid=0x40bc runnable
> [0x00007f8214382000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000521ddf488> (a sun.nio.ch.Util$2)
> - locked <0x0000000521ddf470> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000521c88d38> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-30" prio=10 tid=0x00007f8584326000 nid=0x40bb waiting for
> monitor entry [0x00007f8214482000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
> - waiting to lock <0x0000000519111590> (a
> org.apache.flume.channel.file.LogFileV3$Writer)
>  at org.apache.flume.channel.file.Log.put(Log.java:507)
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
>  at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>  at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>  at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>  at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>  at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-26" prio=10 tid=0x00007f85d82b8000 nid=0x40ba runnable
> [0x00007f8214584000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000521cbcb90> (a sun.nio.ch.Util$2)
> - locked <0x0000000521cbcb78> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000521279250> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-28" prio=10 tid=0x00007f85882ef800 nid=0x40b9 runnable
> [0x00007f8214685000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000521a1c2d0> (a sun.nio.ch.Util$2)
> - locked <0x0000000521a1c2b8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005218f1508> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-34" prio=10 tid=0x00007f8580393800 nid=0x40b8 runnable
> [0x00007f8214786000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052181ce58> (a sun.nio.ch.Util$2)
> - locked <0x000000052181ce40> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005212791c0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-29" prio=10 tid=0x00007f858430b800 nid=0x40b7 runnable
> [0x00007f8214887000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005219175f0> (a sun.nio.ch.Util$2)
> - locked <0x0000000521746090> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000521278e60> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-27" prio=10 tid=0x00007f85882d4800 nid=0x40b6 runnable
> [0x00007f8214988000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052163e9c8> (a sun.nio.ch.Util$2)
> - locked <0x000000052163e9b0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000521279130> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-33" prio=10 tid=0x00007f8580378800 nid=0x40b5 runnable
> [0x00007f8214a89000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052152cb38> (a sun.nio.ch.Util$2)
> - locked <0x000000052152cb20> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000521278d40> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-28" prio=10 tid=0x00007f85842f0800 nid=0x40b4 runnable
> [0x00007f8214b8a000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005213ba3e0> (a sun.nio.ch.Util$2)
> - locked <0x00000005213ba3c8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000521278020> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-27" prio=10 tid=0x00007f85842d5800 nid=0x40b3 runnable
> [0x00007f8214c8b000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005213ed660> (a sun.nio.ch.Util$2)
> - locked <0x00000005213ed648> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000521278c20> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-26" prio=10 tid=0x00007f85882b9800 nid=0x40b2 runnable
> [0x00007f8214d8c000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052133b7c0> (a sun.nio.ch.Util$2)
> - locked <0x000000052133b7a8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005212790a0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-32" prio=10 tid=0x00007f858035d800 nid=0x40af runnable
> [0x00007f8214e8d000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000520e53be0> (a sun.nio.ch.Util$2)
> - locked <0x0000000520e53bc8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000520f3bb50> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-25" prio=10 tid=0x00007f85d829d000 nid=0x40ae runnable
> [0x00007f8214f8e000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000520e09a30> (a sun.nio.ch.Util$2)
> - locked <0x0000000520e09a18> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000520f3b6d0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-31" prio=10 tid=0x00007f8580342800 nid=0x40ad runnable
> [0x00007f821508f000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000520ef70e0> (a sun.nio.ch.Util$2)
> - locked <0x0000000520ef70c8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000520f3b208> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-26" prio=10 tid=0x00007f85842ba800 nid=0x40ac runnable
> [0x00007f8215190000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000520c9dfc0> (a sun.nio.ch.Util$2)
> - locked <0x0000000520c9dfa8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000520bb3cc0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-25" prio=10 tid=0x00007f85842a0000 nid=0x40ab runnable
> [0x00007f8215291000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000520b941a8> (a sun.nio.ch.Util$2)
> - locked <0x0000000520b94190> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000520bb30c0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-24" prio=10 tid=0x00007f85d8282800 nid=0x40aa runnable
> [0x00007f8215392000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000520952288> (a sun.nio.ch.Util$2)
> - locked <0x0000000520952270> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000520356888> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-30" prio=10 tid=0x00007f8580328000 nid=0x40a9 runnable
> [0x00007f8215493000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000520a2fa20> (a sun.nio.ch.Util$2)
> - locked <0x0000000520a2fa08> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000520983da0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-23" prio=10 tid=0x00007f85d8267800 nid=0x40a8 runnable
> [0x00007f8215594000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052095c470> (a sun.nio.ch.Util$2)
> - locked <0x000000052095c458> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000520356840> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-29" prio=10 tid=0x00007f858030d000 nid=0x40a7 runnable
> [0x00007f8215695000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052065b178> (a sun.nio.ch.Util$2)
> - locked <0x000000052065b160> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000520357320> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-28" prio=10 tid=0x00007f85802f2800 nid=0x40a6 runnable
> [0x00007f8215796000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052043cad0> (a sun.nio.ch.Util$2)
> - locked <0x000000052043cab8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005203567b0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-25" prio=10 tid=0x00007f858829f000 nid=0x40a5 runnable
> [0x00007f8215897000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052041a8a8> (a sun.nio.ch.Util$2)
> - locked <0x000000052041a890> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000520356690> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-22" prio=10 tid=0x00007f85d824c800 nid=0x40a4 runnable
> [0x00007f8215998000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052036b380> (a sun.nio.ch.Util$2)
> - locked <0x000000052036b368> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000520357290> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-24" prio=10 tid=0x00007f8584285800 nid=0x40a3 runnable
> [0x00007f8215a99000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000052007ac40> (a sun.nio.ch.Util$2)
> - locked <0x000000052007ac28> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051fe41948> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-27" prio=10 tid=0x00007f85802d8000 nid=0x40a2 runnable
> [0x00007f8215b9a000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005201a2ae8> (a sun.nio.ch.Util$2)
> - locked <0x00000005201a2ad0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ffda458> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-21" prio=10 tid=0x00007f85d8232000 nid=0x40a1 waiting for
> monitor entry [0x00007f8215c9a000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
> - waiting to lock <0x0000000518ee4c90> (a
> org.apache.flume.channel.file.FlumeEventQueue)
>  at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
> at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>  at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>  at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-26" prio=10 tid=0x00007f85802bd800 nid=0x40a0 runnable
> [0x00007f8215d9c000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051ff20e78> (a sun.nio.ch.Util$2)
>  - locked <0x000000051ff20e60> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051fe41900> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-24" prio=10 tid=0x00007f8588284000 nid=0x409f runnable
> [0x00007f8215e9d000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051fbc1ca0> (a sun.nio.ch.Util$2)
>  - locked <0x000000051fbc1c88> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f9a7978> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-25" prio=10 tid=0x00007f85802a2800 nid=0x409e runnable
> [0x00007f8215f9e000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051fa3b5b8> (a sun.nio.ch.Util$2)
>  - locked <0x000000051fa3b5a0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f3155b0> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-20" prio=10 tid=0x00007f85d8217800 nid=0x409d runnable
> [0x00007f821609f000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051f83c838> (a sun.nio.ch.Util$2)
>  - locked <0x000000051f83c820> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f3150e8> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-23" prio=10 tid=0x00007f8588269800 nid=0x409c runnable
> [0x00007f82161a0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051f78b9b0> (a sun.nio.ch.Util$2)
>  - locked <0x000000051f78b998> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f3156d0> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-22" prio=10 tid=0x00007f858824e800 nid=0x409b runnable
> [0x00007f82162a1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051f7c0cc8> (a sun.nio.ch.Util$2)
>  - locked <0x000000051f7c0cb0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f3151c0> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-23" prio=10 tid=0x00007f858426a800 nid=0x409a runnable
> [0x00007f82163a2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051f6065a8> (a sun.nio.ch.Util$2)
>  - locked <0x000000051f606590> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f3d9668> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-22" prio=10 tid=0x00007f858424f800 nid=0x4099 runnable
> [0x00007f82164a3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051f56d710> (a sun.nio.ch.Util$2)
>  - locked <0x000000051f56d6f8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f314f80> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-21" prio=10 tid=0x00007f8588233800 nid=0x4098 runnable
> [0x00007f82165a4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051f3bcd68> (a sun.nio.ch.Util$2)
>  - locked <0x000000051f3bcd50> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f315400> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-19" prio=10 tid=0x00007f85d81fc800 nid=0x4097 runnable
> [0x00007f82166a5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051f53b198> (a sun.nio.ch.Util$2)
>  - locked <0x000000051f53b180> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f315760> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-20" prio=10 tid=0x00007f8588218800 nid=0x4096 waiting for
> monitor entry [0x00007f82167a5000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
>  - waiting to lock <0x0000000519111590> (a
> org.apache.flume.channel.file.LogFileV3$Writer)
> at org.apache.flume.channel.file.Log.put(Log.java:507)
>  at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
> at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>  at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>  at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>  at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>  at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-18" prio=10 tid=0x00007f85d81e1800 nid=0x4095 runnable
> [0x00007f82168a7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051ee92ba0> (a sun.nio.ch.Util$2)
>  - locked <0x000000051ee92b88> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051ed19740> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-19" prio=10 tid=0x00007f85881fd800 nid=0x4094 runnable
> [0x00007f82169a8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051ee5da40> (a sun.nio.ch.Util$2)
>  - locked <0x000000051ee5da28> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051e77a940> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-17" prio=10 tid=0x00007f85d81c6800 nid=0x4093 runnable
> [0x00007f8216aa9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051ecb1730> (a sun.nio.ch.Util$2)
>  - locked <0x000000051ecb1718> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051e77a8b0> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-21" prio=10 tid=0x00007f8584234800 nid=0x4092 runnable
> [0x00007f8216baa000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051e899660> (a sun.nio.ch.Util$2)
>  - locked <0x000000051e899648> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051e77a310> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-24" prio=10 tid=0x00007f8580287000 nid=0x4091 runnable
> [0x00007f8216caa000]
>    java.lang.Thread.State: RUNNABLE
> at
> org.apache.flume.channel.file.FlumeEventQueue.addTail(FlumeEventQueue.java:165)
>  - locked <0x0000000518ee4c90> (a
> org.apache.flume.channel.file.FlumeEventQueue)
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:529)
>  - locked <0x0000000518ee4c90> (a
> org.apache.flume.channel.file.FlumeEventQueue)
> at
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
>  at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>  at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>  at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>  at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-23" prio=10 tid=0x00007f858026c000 nid=0x4090 runnable
> [0x00007f8216dac000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051ea8b6f0> (a sun.nio.ch.Util$2)
> - locked <0x000000051ea8b6d8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051e77a820> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-16" prio=10 tid=0x00007f85d81ac000 nid=0x408f runnable
> [0x00007f8216ead000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051e7c8808> (a sun.nio.ch.Util$2)
> - locked <0x000000051e7c87f0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051e77a280> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-20" prio=10 tid=0x00007f8584219800 nid=0x408e runnable
> [0x00007f8216fae000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051e8fffa8> (a sun.nio.ch.Util$2)
> - locked <0x000000051e8fff90> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051e77a700> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-22" prio=10 tid=0x00007f8580251000 nid=0x408d runnable
> [0x00007f82170af000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051e4dc398> (a sun.nio.ch.Util$2)
> - locked <0x000000051e4dc380> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051dcbeef8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-19" prio=10 tid=0x00007f85841fe800 nid=0x408c runnable
> [0x00007f82171b0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051e221520> (a sun.nio.ch.Util$2)
> - locked <0x000000051e221508> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051e407860> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-18" prio=10 tid=0x00007f85841e3800 nid=0x408b runnable
> [0x00007f82172b1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051e465298> (a sun.nio.ch.Util$2)
> - locked <0x000000051e465280> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051dcbef88> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-18" prio=10 tid=0x00007f85881e3000 nid=0x408a runnable
> [0x00007f82173b2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051dea12e0> (a sun.nio.ch.Util$2)
> - locked <0x000000051dea12c8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051cfd4d78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-15" prio=10 tid=0x00007f85d8191000 nid=0x4089 runnable
> [0x00007f82174b3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051de322b8> (a sun.nio.ch.Util$2)
> - locked <0x000000051de322a0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051dcbec28> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-14" prio=10 tid=0x00007f85d8176800 nid=0x4088 runnable
> [0x00007f82175b4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051dea29a8> (a sun.nio.ch.Util$2)
> - locked <0x000000051dea2990> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051dcbf018> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-21" prio=10 tid=0x00007f8580236800 nid=0x4087 runnable
> [0x00007f82176b5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051dca95d0> (a sun.nio.ch.Util$2)
> - locked <0x000000051dca95b8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051cfd4e08> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-17" prio=10 tid=0x00007f85841c8800 nid=0x4086 runnable
> [0x00007f82177b6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051da8d058> (a sun.nio.ch.Util$2)
> - locked <0x000000051da8d040> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051cd693f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-20" prio=10 tid=0x00007f858021b800 nid=0x4085 runnable
> [0x00007f82178b7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051d95a5b0> (a sun.nio.ch.Util$2)
> - locked <0x000000051d95a598> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051cfd4e98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-16" prio=10 tid=0x00007f85841ad800 nid=0x4083 runnable
> [0x00007f82179b8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051d92a278> (a sun.nio.ch.Util$2)
> - locked <0x000000051d92a260> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051cfd4fb8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-13" prio=10 tid=0x00007f85d815b800 nid=0x4082 runnable
> [0x00007f8217ab9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051d734788> (a sun.nio.ch.Util$2)
> - locked <0x000000051d734770> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051cd69488> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-19" prio=10 tid=0x00007f8580200800 nid=0x4081 runnable
> [0x00007f8217bba000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051d4268e8> (a sun.nio.ch.Util$2)
> - locked <0x000000051d4268d0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051d139810> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-15" prio=10 tid=0x00007f8584192800 nid=0x4080 runnable
> [0x00007f8217cbb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051d4cda88> (a sun.nio.ch.Util$2)
> - locked <0x000000051d4cda70> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051babcf98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-17" prio=10 tid=0x00007f85881c8000 nid=0x407f runnable
> [0x00007f8217dbc000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051d32ffe8> (a sun.nio.ch.Util$2)
> - locked <0x000000051d32ffd0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051cd691b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-14" prio=10 tid=0x00007f8584177800 nid=0x407e runnable
> [0x00007f8217ebd000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051d506c60> (a sun.nio.ch.Util$2)
> - locked <0x000000051d506c48> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051cfd5048> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-16" prio=10 tid=0x00007f85881ad000 nid=0x407d runnable
> [0x00007f8217fbe000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051d1223e0> (a sun.nio.ch.Util$2)
> - locked <0x000000051d1223c8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051c6c7b40> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-15" prio=10 tid=0x00007f8588192800 nid=0x407c runnable
> [0x00007f82180bf000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051d146f50> (a sun.nio.ch.Util$2)
> - locked <0x000000051d146f38> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051cd69248> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-13" prio=10 tid=0x00007f858415c800 nid=0x407b runnable
> [0x00007f82181c0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051d11d1b8> (a sun.nio.ch.Util$2)
> - locked <0x000000051d11d1a0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051c6c7c18> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-12" prio=10 tid=0x00007f85d8141000 nid=0x407a runnable
> [0x00007f82182c1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051cd01a78> (a sun.nio.ch.Util$2)
> - locked <0x000000051cd01a60> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051cd695a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-11" prio=10 tid=0x00007f85d8126800 nid=0x4079 runnable
> [0x00007f82183c2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051cfa6f88> (a sun.nio.ch.Util$2)
> - locked <0x000000051cfa6f70> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ba67370> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-14" prio=10 tid=0x00007f8588177800 nid=0x4078 runnable
> [0x00007f82184c3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051ce5cdd0> (a sun.nio.ch.Util$2)
> - locked <0x000000051ce5cdb8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051c6c7bd0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-12" prio=10 tid=0x00007f8584142000 nid=0x4077 runnable
> [0x00007f82185c4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051cc27cb8> (a sun.nio.ch.Util$2)
> - locked <0x000000051cc27ca0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051babd0b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-18" prio=10 tid=0x00007f85801e6000 nid=0x4076 runnable
> [0x00007f82186c5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051cb895c8> (a sun.nio.ch.Util$2)
> - locked <0x000000051cc77fd0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051c6c7cf0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-11" prio=10 tid=0x00007f8584127800 nid=0x4075 runnable
> [0x00007f82187c6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051ca540d0> (a sun.nio.ch.Util$2)
> - locked <0x000000051caad368> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051c6c7e10> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-13" prio=10 tid=0x00007f858815c800 nid=0x4074 runnable
> [0x00007f82188c7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051c953ab8> (a sun.nio.ch.Util$2)
> - locked <0x000000051ca799a8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051babd028> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-17" prio=10 tid=0x00007f85801cb800 nid=0x4073 runnable
> [0x00007f82189c8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051c77c3d8> (a sun.nio.ch.Util$2)
> - locked <0x000000051c8137c8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051c3856f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-10" prio=10 tid=0x00007f858410c800 nid=0x4072 runnable
> [0x00007f8218ac9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051c7dd2b0> (a sun.nio.ch.Util$2)
> - locked <0x000000051c613988> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051babcde8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-16" prio=10 tid=0x00007f85801b1000 nid=0x4071 runnable
> [0x00007f8218bca000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051c776580> (a sun.nio.ch.Util$2)
> - locked <0x000000051c6633c8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051c6c7ea0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-9" prio=10 tid=0x00007f85840f1800 nid=0x4070 runnable
> [0x00007f8218ccb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051c36b550> (a sun.nio.ch.Util$2)
> - locked <0x000000051c36b568> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ba675f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-8" prio=10 tid=0x00007f85840d6800 nid=0x406f runnable
> [0x00007f8218dcc000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051c546b00> (a sun.nio.ch.Util$2)
> - locked <0x000000051c40a178> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ba67760> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-7" prio=10 tid=0x00007f85840bb800 nid=0x406e runnable
> [0x00007f8218ecd000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051c2ecf90> (a sun.nio.ch.Util$2)
> - locked <0x000000051c29a100> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ba675b0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-15" prio=10 tid=0x00007f8580196000 nid=0x406d runnable
> [0x00007f8218fce000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051c2ee670> (a sun.nio.ch.Util$2)
> - locked <0x000000051c2ee688> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ba67718> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-12" prio=10 tid=0x00007f8588141800 nid=0x406c runnable
> [0x00007f82190cf000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051c0bd978> (a sun.nio.ch.Util$2)
> - locked <0x000000051c16cc00> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051babcf08> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-6" prio=10 tid=0x00007f85840a0800 nid=0x406b runnable
> [0x00007f82191d0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051c068508> (a sun.nio.ch.Util$2)
> - locked <0x000000051bf825f0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ba676d0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-11" prio=10 tid=0x00007f8588127000 nid=0x406a runnable
> [0x00007f82192d1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051becaa68> (a sun.nio.ch.Util$2)
> - locked <0x000000051c03ead0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051babce78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-10" prio=10 tid=0x00007f858810c800 nid=0x4069 runnable
> [0x00007f82193d2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051be31b98> (a sun.nio.ch.Util$2)
> - locked <0x000000051bd56790> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ba67490> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-14" prio=10 tid=0x00007f858017b000 nid=0x4068 runnable
> [0x00007f82194d3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051bccb8c0> (a sun.nio.ch.Util$2)
> - locked <0x000000051bb14420> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051babcd58> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-5" prio=10 tid=0x00007f8584085800 nid=0x4067 runnable
> [0x00007f82195d4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051bd004a8> (a sun.nio.ch.Util$2)
> - locked <0x000000051bd49480> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ba67400> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-4" prio=10 tid=0x00007f858406b000 nid=0x4066 runnable
> [0x00007f82196d5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051ba1e928> (a sun.nio.ch.Util$2)
> - locked <0x000000051bccdd00> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ba66ef0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-9" prio=10 tid=0x00007f85880f1800 nid=0x4065 waiting for
> monitor entry [0x00007f82197d5000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
> - waiting to lock <0x0000000519111590> (a
> org.apache.flume.channel.file.LogFileV3$Writer)
>  at org.apache.flume.channel.file.Log.put(Log.java:507)
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
>  at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>  at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>  at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>  at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>  at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>  at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-8" prio=10 tid=0x00007f85880d6800 nid=0x4064 runnable
> [0x00007f82198d7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051b5bcca8> (a sun.nio.ch.Util$2)
> - locked <0x000000051b7ef8f0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051b5c7cb8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-7" prio=10 tid=0x00007f85880bb800 nid=0x4063 runnable
> [0x00007f82199d8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051b788ce8> (a sun.nio.ch.Util$2)
> - locked <0x000000051b788d00> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051b285260> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-13" prio=10 tid=0x00007f8580160800 nid=0x4062 runnable
> [0x00007f8219ad9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051b71a0f8> (a sun.nio.ch.Util$2)
> - locked <0x000000051b750e98> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051aed5a70> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-10" prio=10 tid=0x00007f85d810b800 nid=0x4061 runnable
> [0x00007f8219bda000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051b517878> (a sun.nio.ch.Util$2)
> - locked <0x000000051b5bab30> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051aed5b90> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-9" prio=10 tid=0x00007f85d80f0800 nid=0x4060 runnable
> [0x00007f8219cdb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051b5656e0> (a sun.nio.ch.Util$2)
> - locked <0x000000051b5657a0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051b285338> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-8" prio=10 tid=0x00007f85d80d5800 nid=0x405f runnable
> [0x00007f8219ddc000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051b3df220> (a sun.nio.ch.Util$2)
> - locked <0x000000051b475118> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051aed5c20> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-6" prio=10 tid=0x00007f85880a0800 nid=0x405e runnable
> [0x00007f8219edd000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051b11c010> (a sun.nio.ch.Util$2)
> - locked <0x000000051b1eab78> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051af91478> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-7" prio=10 tid=0x00007f85d80bb000 nid=0x405d runnable
> [0x00007f8219fde000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051b1d36e0> (a sun.nio.ch.Util$2)
> - locked <0x000000051b0328c0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051aed5cb0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-12" prio=10 tid=0x00007f8580145800 nid=0x405c runnable
> [0x00007f821a0df000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051aeb3070> (a sun.nio.ch.Util$2)
> - locked <0x000000051aeb3088> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a98a820> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-6" prio=10 tid=0x00007f85d80a0000 nid=0x405b runnable
> [0x00007f821a1e0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051aefa920> (a sun.nio.ch.Util$2)
> - locked <0x000000051af3ee40> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ac28810> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-5" prio=10 tid=0x00007f8588086000 nid=0x405a runnable
> [0x00007f821a2e1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051aefe1b0> (a sun.nio.ch.Util$2)
> - locked <0x000000051ad2b858> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051aed5e18> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-11" prio=10 tid=0x00007f858012a800 nid=0x4059 runnable
> [0x00007f821a3e2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051ac42588> (a sun.nio.ch.Util$2)
> - locked <0x000000051ad05d70> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a98a7d8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-3" prio=10 tid=0x00007f8584050000 nid=0x4058 runnable
> [0x00007f821a4e3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051ad06100> (a sun.nio.ch.Util$2)
> - locked <0x000000051ad47428> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051ac289c0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-10" prio=10 tid=0x00007f858010f800 nid=0x4057 runnable
> [0x00007f821a5e4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051aab5d80> (a sun.nio.ch.Util$2)
> - locked <0x000000051ab8ce38> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a98a748> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-5" prio=10 tid=0x00007f85d8085800 nid=0x4056 runnable
> [0x00007f821a6e5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a962980> (a sun.nio.ch.Util$2)
> - locked <0x000000051a775fb0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a842098> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-2" prio=10 tid=0x00007f8584035000 nid=0x4055 runnable
> [0x00007f821a7e6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a817210> (a sun.nio.ch.Util$2)
> - locked <0x000000051a7d2b48> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a841ca8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-4" prio=10 tid=0x00007f85d806b000 nid=0x4054 runnable
> [0x00007f821a8e7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a903340> (a sun.nio.ch.Util$2)
> - locked <0x000000051a903358> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a42db50> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-9" prio=10 tid=0x00007f85800f4800 nid=0x4053 runnable
> [0x00007f821a9e8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a4f17d8> (a sun.nio.ch.Util$2)
> - locked <0x000000051a37ae08> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a22e340> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-4" prio=10 tid=0x00007f858806b000 nid=0x4052 runnable
> [0x00007f821aae9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a4d81e8> (a sun.nio.ch.Util$2)
> - locked <0x000000051a616bb0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a1dfb40> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-3" prio=10 tid=0x00007f85d8050000 nid=0x4051 runnable
> [0x00007f821abea000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a4fb240> (a sun.nio.ch.Util$2)
> - locked <0x000000051a3913b8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a213878> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-2" prio=10 tid=0x00007f85d8035800 nid=0x4050 runnable
> [0x00007f821aceb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a506248> (a sun.nio.ch.Util$2)
> - locked <0x000000051a506308> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a42dd48> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-8" prio=10 tid=0x00007f85800d9800 nid=0x404f runnable
> [0x00007f821adec000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a2d5170> (a sun.nio.ch.Util$2)
> - locked <0x000000051a2d5188> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a22e2b0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-7" prio=10 tid=0x00007f85800be800 nid=0x404e runnable
> [0x00007f821aeed000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a2d3cb8> (a sun.nio.ch.Util$2)
> - locked <0x000000051a2b8310> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a213758> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
> [0x00007f821afee000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
> - waiting to lock <0x0000000518ee4c90> (a
> org.apache.flume.channel.file.FlumeEventQueue)
>  at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
> at
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>  at
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>  at
> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f836c01a000
> nid=0x404c runnable [0x00007f821b0ef000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a045bb0> (a sun.nio.ch.Util$2)
> - locked <0x000000051a045bc8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a000c98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b376800
> nid=0x404b waiting on condition [0x00007f821b1f0000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051a045c10> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.NativeThread.current(Native Method)
> at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>  at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
> - locked <0x00000005190ec998> (a java.lang.Object)
>  at org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
> at org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>  - locked <0x0000000519111590> (a
> org.apache.flume.channel.file.LogFileV3$Writer)
> at org.apache.flume.channel.file.Log.take(Log.java:550)
>  at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
> at
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>  at
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>  at
> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8354035000
> nid=0x4049 runnable [0x00007f821b3f2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a045c28> (a sun.nio.ch.Util$2)
> - locked <0x000000051a045c40> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a000ce0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834801a000
> nid=0x4048 runnable [0x00007f821b4f3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a17bd60> (a sun.nio.ch.Util$2)
> - locked <0x000000051a17bd78> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a180230> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b374800
> nid=0x4047 waiting on condition [0x00007f821b5f4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051a08ba88> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857338a800
> nid=0x4046 waiting on condition [0x00007f821b6f5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051a147940> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
> tid=0x00007f85f3bff800 nid=0x4045 waiting on condition [0x00007f821b7f6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000056dca65c0> (a
> java.util.concurrent.FutureTask$Sync)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
>  at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:227)
>  at java.util.concurrent.FutureTask.get(FutureTask.java:91)
> at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:323)
>  at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:269)
> at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:352)
>  at
> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8344035000
> nid=0x4044 runnable [0x00007f821b8f7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a184a90> (a sun.nio.ch.Util$2)
> - locked <0x000000051a184aa8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a0bb2e0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bfd800
> nid=0x4043 waiting on condition [0x00007f821b9f8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051a147958> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f833001a000
> nid=0x4042 runnable [0x00007f821baf9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a188f28> (a sun.nio.ch.Util$2)
> - locked <0x000000051a188f40> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519d90758> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8334035000
> nid=0x4041 runnable [0x00007f821bbfa000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a045cd0> (a sun.nio.ch.Util$2)
> - locked <0x000000051a045ce8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a000d28> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b372800
> nid=0x4040 waiting on condition [0x00007f821bcfb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f74590> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573388800
> nid=0x403f waiting on condition [0x00007f821bdfc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fb0c90> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832c035000
> nid=0x403e runnable [0x00007f821befd000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051a045d30> (a sun.nio.ch.Util$2)
> - locked <0x000000051a045d48> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051a000d70> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bfb800
> nid=0x403d waiting on condition [0x00007f821bffe000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519e5d388> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f831801a000
> nid=0x403c runnable [0x00007f82341c0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519ebdb70> (a sun.nio.ch.Util$2)
> - locked <0x0000000519ebdb88> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519d3a778> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f831001a000
> nid=0x403b runnable [0x00007f82342c1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519f47f90> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f47fa8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519e11ca8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaf0000
> nid=0x403a waiting on condition [0x00007f82343c2000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f73018> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8308035000
> nid=0x4039 runnable [0x00007f82344c3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519f745a8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f745c0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519ca9f78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573386800
> nid=0x4038 waiting on condition [0x00007f82345c4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fa5ce0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bf9800
> nid=0x4037 waiting on condition [0x00007f82346c5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fb0ca8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8300035000
> nid=0x4036 runnable [0x00007f82347c6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519f22d20> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f22d38> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519d9f4b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaee000
> nid=0x4035 waiting on condition [0x00007f82348c7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fb0cc0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82fc01a000
> nid=0x4034 runnable [0x00007f82349c8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519f74608> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f74620> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519ca9fc0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82f001a000
> nid=0x4033 runnable [0x00007f8234ac9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519fb0cd8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fb0cf0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519cadd40> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f44800
> nid=0x4032 waiting on condition [0x00007f8234bca000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519ebdbd0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3377000
> nid=0x4031 waiting on condition [0x00007f8234ccb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f701c0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82e8035000
> nid=0x4030 runnable [0x00007f8234dcc000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519f47ff0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f48008> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519e11cf0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaec000
> nid=0x402f waiting on condition [0x00007f8234ecd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fa5cf8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
> tid=0x00007f857fc38000 nid=0x402e waiting for monitor entry
> [0x00007f8234fce000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
> - waiting to lock <0x0000000518ee4c90> (a
> org.apache.flume.channel.file.FlumeEventQueue)
>  at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
> at
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>  at
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>  at
> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82dc01a000
> nid=0x402d runnable [0x00007f82350cf000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519ebdbe8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519ebdc00> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519d3a7c0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82d001a000
> nid=0x402c runnable [0x00007f82351d0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519fb0d38> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fb0d50> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519cadd88> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-144-thread-1" prio=10 tid=0x00007f85d8001000 nid=0x402b runnable
> [0x00007f82352d1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519f481a0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f49040> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519e11d80> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6d3800
> nid=0x402a waiting on condition [0x00007f82353d2000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f22d80> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f42800
> nid=0x4029 waiting on condition [0x00007f82354d3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fa5d10> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82cc035000
> nid=0x4028 runnable [0x00007f82355d4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519fb0d98> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fb0db0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519caddd0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-3" prio=10 tid=0x00007f8588001000 nid=0x4027 runnable
> [0x00007f82356d5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519ebdc48> (a sun.nio.ch.Util$2)
> - locked <0x0000000519ebdc60> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519d3a808> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3375000
> nid=0x4026 waiting on condition [0x00007f82357d6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fa5d28> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82c4035000
> nid=0x4025 runnable [0x00007f82358d7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519f22d98> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f22db0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519d9f500> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaea000
> nid=0x4024 waiting on condition [0x00007f82359d8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fa5d40> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a8035000
> nid=0x4023 runnable [0x00007f8235ad9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519fb0df8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fb0e10> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519cade18> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82b401a000
> nid=0x4022 runnable [0x00007f8235bda000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519f73030> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f73048> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519e69de8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82ac035000
> nid=0x4021 runnable [0x00007f8235cdb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519f61380> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f61398> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005197600f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6d1800
> nid=0x4020 waiting on condition [0x00007f8235ddc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f22df8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f40800
> nid=0x401f waiting on condition [0x00007f8235edd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fa5d58> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3373000
> nid=0x401e waiting on condition [0x00007f8235fde000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519ebdca8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a4035000
> nid=0x401d runnable [0x00007f82360df000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519fb0e58> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fb0e70> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519cade60> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a8000
> nid=0x401c waiting on condition [0x00007f82361e0000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f22e10> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-6" prio=10 tid=0x00007f85800a3800 nid=0x401b runnable
> [0x00007f82362e1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519ebde28> (a sun.nio.ch.Util$2)
> - locked <0x0000000519de7350> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519d3a898> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828801a000
> nid=0x401a runnable [0x00007f82363e2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519fa5d70> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fa5d88> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519d36338> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828c035000
> nid=0x4019 runnable [0x00007f82364e3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519f73090> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f730a8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519e69e30> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8280035000
> nid=0x4018 runnable [0x00007f82365e4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519fb0eb8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519e1a7f8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519cadea8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6cf800
> nid=0x4017 waiting on condition [0x00007f82366e5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f22e28> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f3e800
> nid=0x4016 waiting on condition [0x00007f82367e6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f701d8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3371000
> nid=0x4015 waiting on condition [0x00007f82368e7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fb0f00> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8278035000
> nid=0x4014 runnable [0x00007f82369e8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519e93628> (a sun.nio.ch.Util$2)
> - locked <0x0000000519e93640> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519d9f548> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a6000
> nid=0x4013 waiting on condition [0x00007f8236ae9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005199bdaf0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f827001a000
> nid=0x4012 runnable [0x00007f8236bea000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519fb0f18> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fb0f30> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519cadef0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6cd800
> nid=0x4011 waiting on condition [0x00007f8236ceb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c47f38> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826c035000
> nid=0x4010 runnable [0x00007f8236dec000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519fa5dd0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fa5de8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519d36380> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826001a000
> nid=0x400f runnable [0x00007f8236eed000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c8c170> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c8c188> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519a73d78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572aeb000
> nid=0x400e waiting on condition [0x00007f8236fee000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c7e570> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2e800
> nid=0x400d waiting on condition [0x00007f82370ef000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c47f50> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8258035000
> nid=0x400b runnable [0x00007f82371f0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c4b190> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c4b1a8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005199eb068> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a4000
> nid=0x400a waiting on condition [0x00007f82372f1000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c47f68> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f825001a000
> nid=0x4009 runnable [0x00007f82373f2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c33550> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c33568> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051996b7b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e29c000
> nid=0x4008 waiting on condition [0x00007f82374f3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c47f80> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f824c03f800
> nid=0x4007 runnable [0x00007f82375f4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c7e588> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e5a0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005199a1910> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8240035000
> nid=0x4006 runnable [0x00007f82376f5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c47f98> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c47fb0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005199536b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae9000
> nid=0x4005 waiting on condition [0x00007f82377f6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c368c8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2c800
> nid=0x4004 waiting on condition [0x00007f82378f7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c29bf0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8238035000
> nid=0x4003 runnable [0x00007f82379f8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519bd8198> (a sun.nio.ch.Util$2)
> - locked <0x0000000519bd81b0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519b59da0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a2000
> nid=0x4002 waiting on condition [0x00007f8237af9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c7e5e8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8230035000
> nid=0x4001 runnable [0x00007f8237bfa000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c47ff8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c48010> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519953700> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e29a000
> nid=0x4000 waiting on condition [0x00007f8237cfb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c7e600> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f823001a000
> nid=0x3fff runnable [0x00007f8237dfc000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c7e618> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e630> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005199a1958> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f823c01a000
> nid=0x3ffe runnable [0x00007f8237efd000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c48058> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c48070> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519a6d798> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f823801a000
> nid=0x3ffd runnable [0x00007f8237ffe000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c335b0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c335c8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051996b800> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae7000
> nid=0x3ffc waiting on condition [0x00007f84001c0000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c480b8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2a800
> nid=0x3ffb waiting on condition [0x00007f84002c1000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519bd81f8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24e800
> nid=0x3ffa waiting on condition [0x00007f84003c2000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c29c08> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f824001a000
> nid=0x3ff9 runnable [0x00007f84004c3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c7e678> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e690> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005199a19a0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e298000
> nid=0x3ff8 waiting on condition [0x00007f84005c4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c67408> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-5" prio=10 tid=0x00007f8580088800 nid=0x3ff7 runnable
> [0x00007f84006c5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c48238> (a sun.nio.ch.Util$2)
> - locked <0x0000000519d5e0d0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519a6d828> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f825c01a000
> nid=0x3ff6 runnable [0x00007f84007c6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c368e0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c368f8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519953088> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f825801a000
> nid=0x3ff5 runnable [0x00007f84008c7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c33610> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c33628> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051996b848> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826401a000
> nid=0x3ff4 runnable [0x00007f84009c8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c7e6d8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e6f0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005199a19e8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae5000
> nid=0x3ff3 waiting on condition [0x00007f8400ac9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c730b8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24c800
> nid=0x3ff2 waiting on condition [0x00007f8400bca000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c29c20> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f29000
> nid=0x3ff1 waiting on condition [0x00007f8400ccb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c4b1f0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826c01a000
> nid=0x3ff0 runnable [0x00007f8400dcc000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519bd8210> (a sun.nio.ch.Util$2)
> - locked <0x0000000519bd8228> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519b59de8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e296800
> nid=0x3fef waiting on condition [0x00007f8400ecd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c8c1d0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f827801a000
> nid=0x3fee runnable [0x00007f8400fce000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c7e738> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e750> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005199a1a30> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828401a000
> nid=0x3fed runnable [0x00007f84010cf000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c33670> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c33688> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051996b890> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828001a000
> nid=0x3fec runnable [0x00007f84011d0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c29c38> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c29c50> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519955c68> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828c01a000
> nid=0x3feb runnable [0x00007f84012d1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c7e798> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e7b0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005199533a0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae6000
> nid=0x3fea waiting on condition [0x00007f84013d2000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519bd8270> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24a800
> nid=0x3fe9 waiting on condition [0x00007f84014d3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c36940> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572273800
> nid=0x3fe8 waiting on condition [0x00007f84015d4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c8c1e8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de54800
> nid=0x3fe7 waiting on condition [0x00007f84016d5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c67420> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f829001a000
> nid=0x3fe6 runnable [0x00007f84017d6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c7e7f8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e810> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005199533e8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a401a000
> nid=0x3fe5 runnable [0x00007f84018d7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c336d0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c336e8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051996b8d8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae4000
> nid=0x3fe4 waiting on condition [0x00007f84019d8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519946350> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-4" prio=10 tid=0x00007f858006d800 nid=0x3fe3 runnable
> [0x00007f8401ad9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519bd83d8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f042a0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519b59e78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82ac01a000
> nid=0x3fe2 runnable [0x00007f8401bda000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c7e858> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e870> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519953430> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a248800
> nid=0x3fe1 waiting on condition [0x00007f8401cdb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519890ec8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a801a000
> nid=0x3fe0 runnable [0x00007f8401ddc000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519c8c200> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c8c218> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519a73dc0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572271800
> nid=0x3fdf waiting on condition [0x00007f8401edd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198a3170> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de52800
> nid=0x3fde waiting on condition [0x00007f8401fde000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198b7610> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82b801a000
> nid=0x3fdd runnable [0x00007f84020df000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519947f70> (a sun.nio.ch.Util$2)
> - locked <0x0000000519947f88> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005197bc938> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82c401a000
> nid=0x3fdc runnable [0x00007f84021e0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519946368> (a sun.nio.ch.Util$2)
> - locked <0x0000000519946380> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051983b9b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82cc01a000
> nid=0x3fdb runnable [0x00007f84022e1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005198a3188> (a sun.nio.ch.Util$2)
> - locked <0x00000005198a31a0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005197300e0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae2000
> nid=0x3fda waiting on condition [0x00007f84023e2000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005199463c8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857226f800
> nid=0x3fd9 waiting on condition [0x00007f84024e3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519792380> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82c801a000
> nid=0x3fd8 runnable [0x00007f84025e4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519947fd0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519947fe8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005197bc980> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579597000
> nid=0x3fd7 waiting on condition [0x00007f84026e5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519488cc0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de50800
> nid=0x3fd6 waiting on condition [0x00007f84027e6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198cc640> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82d801a000
> nid=0x3fd5 runnable [0x00007f84028e7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005199463e0> (a sun.nio.ch.Util$2)
> - locked <0x00000005199463f8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051983ba00> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82e401a000
> nid=0x3fd4 runnable [0x00007f84029e8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005198f7fc8> (a sun.nio.ch.Util$2)
> - locked <0x00000005198f7fe0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194d0160> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82ec01a000
> nid=0x3fd3 runnable [0x00007f8402ae9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005198a31e8> (a sun.nio.ch.Util$2)
> - locked <0x00000005198a3200> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519730128> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae0800
> nid=0x3fd2 waiting on condition [0x00007f8402bea000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519948030> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857226d800
> nid=0x3fd1 waiting on condition [0x00007f8402ceb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519946440> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82e801a000
> nid=0x3fd0 runnable [0x00007f8402dec000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519890ee0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519890ef8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005196f56f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579595000
> nid=0x3fcf waiting on condition [0x00007f8402eed000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198b7628> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de4e800
> nid=0x3fce waiting on condition [0x00007f8402fee000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198a3248> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830401a000
> nid=0x3fcd runnable [0x00007f84030ef000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519946458> (a sun.nio.ch.Util$2)
> - locked <0x0000000519946470> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051983ba48> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830001a000
> nid=0x3fcc runnable [0x00007f84031f0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519937b88> (a sun.nio.ch.Util$2)
> - locked <0x0000000519937ba0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519887cb8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830c01a000
> nid=0x3fcb runnable [0x00007f84032f1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519948048> (a sun.nio.ch.Util$2)
> - locked <0x0000000519948060> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005197bc9c8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830801a000
> nid=0x3fca runnable [0x00007f84033f2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005199464b8> (a sun.nio.ch.Util$2)
> - locked <0x00000005199464d0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051983ba90> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e2b000
> nid=0x3fc9 waiting on condition [0x00007f84034f3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519943980> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1595000
> nid=0x3fc8 waiting on condition [0x00007f84035f4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198a3260> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857bff5800
> nid=0x3fc7 waiting on condition [0x00007f84036f5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519901138> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da0c000
> nid=0x3fc6 waiting on condition [0x00007f84037f6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005199480a8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832001a000
> nid=0x3fc5 runnable [0x00007f84038f7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519946518> (a sun.nio.ch.Util$2)
> - locked <0x0000000519946530> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051983bad8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832c01a000
> nid=0x3fc4 runnable [0x00007f84039f8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005198a3278> (a sun.nio.ch.Util$2)
> - locked <0x00000005198a3290> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519730170> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832801a000
> nid=0x3fc3 runnable [0x00007f8403af9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005198f8028> (a sun.nio.ch.Util$2)
> - locked <0x00000005198f8040> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194d01a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f833401a000
> nid=0x3fc2 runnable [0x00007f8403bfa000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005198b7640> (a sun.nio.ch.Util$2)
> - locked <0x00000005198b7658> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005196f3480> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e29000
> nid=0x3fc1 waiting on condition [0x00007f8403cfb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519946578> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1593000
> nid=0x3fc0 waiting on condition [0x00007f8403dfc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005199480c0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579592000
> nid=0x3fbf waiting on condition [0x00007f8403efd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519943998> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da0a800
> nid=0x3fbe waiting on condition [0x00007f84401c0000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519946590> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834401a000
> nid=0x3fbd runnable [0x00007f84402c1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519903410> (a sun.nio.ch.Util$2)
> - locked <0x0000000519903428> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005191a1498> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834001a000
> nid=0x3fbc runnable [0x00007f84403c2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519943ef0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519943f08> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005196f5740> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834c01a000
> nid=0x3fbb runnable [0x00007f84404c3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005199497b0> (a sun.nio.ch.Util$2)
> - locked <0x00000005199497c8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519685f60> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573ff1800
> nid=0x3fba waiting on condition [0x00007f84405c4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198f8088> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1591800
> nid=0x3fb9 waiting on condition [0x00007f84406c5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198a32d8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579591000
> nid=0x3fb8 waiting on condition [0x00007f84407c6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005199480d8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f835401a000
> nid=0x3fb7 runnable [0x00007f84408c7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005199465a8> (a sun.nio.ch.Util$2)
> - locked <0x00000005199465c0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051983bb20> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da08800
> nid=0x3fb6 waiting on condition [0x00007f84409c8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005199439b0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-142-thread-1" prio=10 tid=0x00007f8584001000 nid=0x3fb5 runnable
> [0x00007f8440ac9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005198b77f0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519a00960> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519952e48> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f837401a000
> nid=0x3fb4 runnable [0x00007f8440bca000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519937be8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519937c00> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519887d00> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f836801a000
> nid=0x3fb3 runnable [0x00007f8440ccb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519943f50> (a sun.nio.ch.Util$2)
> - locked <0x0000000519943f68> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005196f5788> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f837001a000
> nid=0x3fb2 runnable [0x00007f8440dcc000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519901150> (a sun.nio.ch.Util$2)
> - locked <0x0000000519901168> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519106100> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f158f800
> nid=0x3fb1 waiting on condition [0x00007f8440ecd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519946608> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e26800
> nid=0x3fb0 waiting on condition [0x00007f8440fce000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198a32f0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913e000
> nid=0x3faf waiting on condition [0x00007f84410cf000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005199439c8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f837801a000
> nid=0x3fae runnable [0x00007f84411d0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005199480f0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519948108> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005197bca10> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da07000
> nid=0x3fad waiting on condition [0x00007f84412d1000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519943fb0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f838801a000
> nid=0x3fac runnable [0x00007f84413d2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005198f80a0> (a sun.nio.ch.Util$2)
> - locked <0x00000005198f80b8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194d01f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f839401a000
> nid=0x3fab runnable [0x00007f84414d3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519946620> (a sun.nio.ch.Util$2)
> - locked <0x0000000519946638> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051983bb68> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f839001a000
> nid=0x3faa runnable [0x00007f84415d4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519937c48> (a sun.nio.ch.Util$2)
> - locked <0x0000000519937c60> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519887d48> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d0d000
> nid=0x3fa9 waiting on condition [0x00007f84416d5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519654310> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913c000
> nid=0x3fa8 waiting on condition [0x00007f84417d6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191b54d8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571164000
> nid=0x3fa7 waiting on condition [0x00007f84418d7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519637a20> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83a801a000
> nid=0x3fa6 runnable [0x00007f84419d8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005195f76c8> (a sun.nio.ch.Util$2)
> - locked <0x00000005196430c0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194ecaf0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83b401a000
> nid=0x3fa5 runnable [0x00007f8441ad9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051965f960> (a sun.nio.ch.Util$2)
> - locked <0x000000051965f978> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519104648> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83b001a000
> nid=0x3fa4 runnable [0x00007f8441bda000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519655e28> (a sun.nio.ch.Util$2)
> - locked <0x0000000519655e40> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519104ba0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-3" prio=10 tid=0x00007f8580052800 nid=0x3fa3 runnable
> [0x00007f8441cdb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519637b10> (a sun.nio.ch.Util$2)
> - locked <0x0000000519694210> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051916e518> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83bc01a000
> nid=0x3fa2 runnable [0x00007f8441ddc000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519597678> (a sun.nio.ch.Util$2)
> - locked <0x0000000519597690> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519106388> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c4800
> nid=0x3fa1 waiting on condition [0x00007f8441edd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519654328> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d0b000
> nid=0x3fa0 waiting on condition [0x00007f8441fde000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519676ad0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571162000
> nid=0x3f9f waiting on condition [0x00007f84420df000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519655618> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913a000
> nid=0x3f9e waiting on condition [0x00007f84421e0000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005194ae248> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-2" prio=10 tid=0x00007f8580001000 nid=0x3f9d runnable
> [0x00007f84422e1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519654340> (a sun.nio.ch.Util$2)
> - locked <0x0000000519654358> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519105380> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83cc01a000
> nid=0x3f9c runnable [0x00007f84423e2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519655e88> (a sun.nio.ch.Util$2)
> - locked <0x0000000519655ea0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519104be8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83d001a000
> nid=0x3f9b runnable [0x00007f84424e3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005194e1350> (a sun.nio.ch.Util$2)
> - locked <0x00000005194e1368> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005191041c8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c3000
> nid=0x3f9a waiting on condition [0x00007f84425e4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051967caf8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83d801a000
> nid=0x3f99 runnable [0x00007f84426e5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519655ee8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519655f00> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519104c30> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83dc01a000
> nid=0x3f98 runnable [0x00007f84427e6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005196543a0> (a sun.nio.ch.Util$2)
> - locked <0x00000005196543b8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005191053c8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d09800
> nid=0x3f97 waiting on condition [0x00007f84428e7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519655630> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579138000
> nid=0x3f96 waiting on condition [0x00007f84429e8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005196592c8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571160000
> nid=0x3f95 waiting on condition [0x00007f8442ae9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519655f48> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f840801a000
> nid=0x3f94 runnable [0x00007f8442bea000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519655648> (a sun.nio.ch.Util$2)
> - locked <0x0000000519655660> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051914b290> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83e801a000
> nid=0x3f93 runnable [0x00007f8442ceb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519643108> (a sun.nio.ch.Util$2)
> - locked <0x0000000519643120> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194ecb38> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d07800
> nid=0x3f92 waiting on condition [0x00007f8442dec000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519658428> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83f801a000
> nid=0x3f91 runnable [0x00007f8442eed000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005194c9908> (a sun.nio.ch.Util$2)
> - locked <0x00000005194c9920> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519511968> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83fc01a000
> nid=0x3f90 runnable [0x00007f8442fee000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519643168> (a sun.nio.ch.Util$2)
> - locked <0x0000000519643180> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194ecb80> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578465000
> nid=0x3f8f waiting on condition [0x00007f8403ffe000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519654400> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857115e800
> nid=0x3f8e waiting on condition [0x00007f84430ef000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005196556a8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c1000
> nid=0x3f8d waiting on condition [0x00007f84431f0000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519654418> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f840c01a000
> nid=0x3f8c runnable [0x00007f84432f1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005196431c8> (a sun.nio.ch.Util$2)
> - locked <0x00000005196431e0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194ecbc8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f841801a000
> nid=0x3f8b runnable [0x00007f84433f2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519655f60> (a sun.nio.ch.Util$2)
> - locked <0x0000000519655f78> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519104c78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b4000
> nid=0x3f8a waiting on condition [0x00007f84434f3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005196721e8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5bf000
> nid=0x3f89 waiting on condition [0x00007f84435f4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519654430> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f841c01a000
> nid=0x3f88 runnable [0x00007f84436f5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005196556c0> (a sun.nio.ch.Util$2)
> - locked <0x00000005196556d8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051914b2d8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f842801a000
> nid=0x3f87 runnable [0x00007f84437f6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519643228> (a sun.nio.ch.Util$2)
> - locked <0x0000000519643240> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194ecc10> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b2000
> nid=0x3f86 waiting on condition [0x00007f84438f7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519654448> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5bd800
> nid=0x3f85 waiting on condition [0x00007f84439f8000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051967cb10> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f843001a000
> nid=0x3f84 runnable [0x00007f8443af9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519655fc0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519655fd8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519104cc0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b0800
> nid=0x3f83 waiting on condition [0x00007f8443bfa000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005196592e0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f843801a000
> nid=0x3f82 runnable [0x00007f8443cfb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519643288> (a sun.nio.ch.Util$2)
> - locked <0x00000005196432a0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194ecc58> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d17b000
> nid=0x3f81 waiting on condition [0x00007f8443dfc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051921afd8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f843c01a000
> nid=0x3f80 runnable [0x00007f8443efd000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519654460> (a sun.nio.ch.Util$2)
> - locked <0x0000000519654478> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519105410> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08ae800
> nid=0x3f7f waiting on condition [0x00007f8443ffe000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519676ae8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f844401a000
> nid=0x3f7e runnable [0x00007f85441a6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051967cb28> (a sun.nio.ch.Util$2)
> - locked <0x000000051967cb40> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194745e0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d179000
> nid=0x3f7d waiting on condition [0x00007f85442a7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005196432e8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f845801a000
> nid=0x3f7c runnable [0x00007f85443a8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519658440> (a sun.nio.ch.Util$2)
> - locked <0x00000005196583f8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519349128> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f845401a000
> nid=0x3f7b runnable [0x00007f85444a9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519676aa0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519676ab8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051921b020> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08ad800
> nid=0x3f7a waiting on condition [0x00007f85446c5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519659298> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578463000
> nid=0x3f79 waiting on condition [0x00007f85447c6000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005194a3568> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f845c01a000
> nid=0x3f78 runnable [0x00007f85445aa000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005196544c0> (a sun.nio.ch.Util$2)
> - locked <0x00000005196544d8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519105458> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d177000
> nid=0x3f77 waiting on condition [0x00007f85448c7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519656020> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f846401a000
> nid=0x3f76 runnable [0x00007f85449c8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519655720> (a sun.nio.ch.Util$2)
> - locked <0x0000000519655738> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519105f98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857115c800
> nid=0x3f75 waiting on condition [0x00007f8544ac9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519402f58> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f847801a000
> nid=0x3f74 runnable [0x00007f8544bca000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519654520> (a sun.nio.ch.Util$2)
> - locked <0x0000000519654538> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005191054a0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f847401a000
> nid=0x3f73 runnable [0x00007f8544ccb000]
>    java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.newKeyIterator(HashMap.java:840)
> at java.util.HashMap$KeySet.iterator(HashMap.java:874)
>  at java.util.HashSet.iterator(HashSet.java:153)
> at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
>  - locked <0x0000000519656248> (a java.util.HashSet)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)
>  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519656038> (a sun.nio.ch.Util$2)
>  - locked <0x0000000519656050> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519104d08> (a sun.nio.ch.EPollSelectorImpl)
>  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>  at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0015800
> nid=0x3f72 waiting on condition [0x00007f8544dcc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>  at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051941d010> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578461000
> nid=0x3f71 waiting on condition [0x00007f8544ecd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519477938> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f847c01a000
> nid=0x3f70 runnable [0x00007f8544fce000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051942aa08> (a sun.nio.ch.Util$2)
> - locked <0x000000051942aa20> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519349050> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d175800
> nid=0x3f6f waiting on condition [0x00007f85450cf000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519470c60> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f848401a000
> nid=0x3f6e runnable [0x00007f85451d0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519445fa0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519445fb8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194679d0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570456800
> nid=0x3f6d waiting on condition [0x00007f85452d1000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519419230> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f849801a000
> nid=0x3f6c runnable [0x00007f85453d2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519470c78> (a sun.nio.ch.Util$2)
> - locked <0x0000000519470c90> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194743a0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f849401a000
> nid=0x3f6b runnable [0x00007f85454d3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051945beb8> (a sun.nio.ch.Util$2)
> - locked <0x000000051945bed0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519196000> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0013800
> nid=0x3f6a waiting on condition [0x00007f85455d4000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519470cd8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857845f000
> nid=0x3f69 waiting on condition [0x00007f85456d5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051941d1c0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f849c01a000
> nid=0x3f68 runnable [0x00007f85457d6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005194583b8> (a sun.nio.ch.Util$2)
> - locked <0x00000005194583d0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194743e8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c488000
> nid=0x3f67 waiting on condition [0x00007f85458d7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519424e80> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84a401a000
> nid=0x3f66 runnable [0x00007f85459d8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051941d1d8> (a sun.nio.ch.Util$2)
> - locked <0x000000051941d1f0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519467a18> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570455000
> nid=0x3f65 waiting on condition [0x00007f8545ad9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051945bf18> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84b801a000
> nid=0x3f64 runnable [0x00007f8545bda000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519419248> (a sun.nio.ch.Util$2)
> - locked <0x0000000519419200> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519431ee0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84c001a000
> nid=0x3f63 runnable [0x00007f8545cdb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051941d238> (a sun.nio.ch.Util$2)
> - locked <0x000000051941d250> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519467a60> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0012000
> nid=0x3f62 waiting on condition [0x00007f8545ddc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519458418> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84bc01a000
> nid=0x3f61 runnable [0x00007f8545edd000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051945bf30> (a sun.nio.ch.Util$2)
> - locked <0x000000051945bf48> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519196048> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857845e000
> nid=0x3f60 waiting on condition [0x00007f8545fde000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519479f10> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c486000
> nid=0x3f5f waiting on condition [0x00007f85460df000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051941d298> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84c401a000
> nid=0x3f5e runnable [0x00007f85461e0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051942a9a8> (a sun.nio.ch.Util$2)
> - locked <0x000000051942a9c0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519349098> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570453800
> nid=0x3f5d waiting on condition [0x00007f85462e1000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051945bf90> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84cc01a000
> nid=0x3f5c runnable [0x00007f85463e2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519458430> (a sun.nio.ch.Util$2)
> - locked <0x0000000519458448> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519474430> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c484800
> nid=0x3f5b waiting on condition [0x00007f85464e3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051941d2b0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84d401a000
> nid=0x3f5a runnable [0x00007f85465e4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005194191e8> (a sun.nio.ch.Util$2)
> - locked <0x000000051936a4e0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519431f28> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570450800
> nid=0x3f59 waiting on condition [0x00007f85466e5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051943ef60> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84dc01a000
> nid=0x3f58 runnable [0x00007f85467e6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051942a948> (a sun.nio.ch.Util$2)
> - locked <0x000000051942a960> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005193490e0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c482800
> nid=0x3f57 waiting on condition [0x00007f85468e7000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051945bfa8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84e401a000
> nid=0x3f56 runnable [0x00007f85469e8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051941d2c8> (a sun.nio.ch.Util$2)
> - locked <0x000000051941d2e0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519467aa8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857044f000
> nid=0x3f55 waiting on condition [0x00007f8546ae9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051945bfc0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84f801a000
> nid=0x3f54 runnable [0x00007f8546bea000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051936a4c8> (a sun.nio.ch.Util$2)
> - locked <0x000000051936a480> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519431f70> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84f401a000
> nid=0x3f53 runnable [0x00007f8546ceb000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519458490> (a sun.nio.ch.Util$2)
> - locked <0x00000005194584a8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519474478> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578008000
> nid=0x3f52 waiting on condition [0x00007f8546dec000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051942a8e8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c00d000
> nid=0x3f51 waiting on condition [0x00007f8546eed000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519447098> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84fc01a000
> nid=0x3f50 runnable [0x00007f8546fee000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051941d328> (a sun.nio.ch.Util$2)
> - locked <0x000000051941d340> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519467af0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857044d000
> nid=0x3f4f waiting on condition [0x00007f85470ef000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051945bfd8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f851001a000
> nid=0x3f4e runnable [0x00007f85471f0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519424e98> (a sun.nio.ch.Util$2)
> - locked <0x0000000519424e50> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519380d20> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f850c01a000
> nid=0x3f4d runnable [0x00007f85472f1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005194584f0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519458508> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005194744c0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578006800
> nid=0x3f4c waiting on condition [0x00007f85473f2000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051945bff0> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c01d800
> nid=0x3f4b waiting on condition [0x00007f85474f3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051936a468> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f851401a000
> nid=0x3f4a runnable [0x00007f85475f4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051941d388> (a sun.nio.ch.Util$2)
> - locked <0x000000051941d3a0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519467b38> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857045b800
> nid=0x3f49 waiting on condition [0x00007f85476f5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051942a900> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f853801a000
> nid=0x3f48 runnable [0x00007f85477f6000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519479f28> (a sun.nio.ch.Util$2)
> - locked <0x0000000519479ee0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519203068> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f853001f800
> nid=0x3f47 runnable [0x00007f85478f7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519477950> (a sun.nio.ch.Util$2)
> - locked <0x0000000519477968> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051922faf0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f852c01f000
> nid=0x3f46 runnable [0x00007f85479f8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005194353f8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519435410> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x000000051921b0f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f853401a000
> nid=0x3f45 runnable [0x00007f8547af9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051945c008> (a sun.nio.ch.Util$2)
> - locked <0x000000051945c020> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000519196090> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578005800
> nid=0x3f44 waiting on condition [0x00007f8547bfa000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005190de788> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f001a800
> nid=0x3f43 waiting on condition [0x00007f8547cfb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518fae040> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c022000
> nid=0x3f42 waiting on condition [0x00007f8547dfc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518cafb20> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570460000
> nid=0x3f41 waiting on condition [0x00007f8547efd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005190db9c8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>  at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-2" prio=10 tid=0x00007f8588036000 nid=0x3f40 runnable
> [0x00007f8547ffe000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005190ee0e8> (a sun.nio.ch.Util$2)
> - locked <0x00000005191f0be8> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005190e1d18> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-140-thread-1" prio=10 tid=0x00007f858801b800 nid=0x3f3f runnable
> [0x00007f858c18a000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000518f16658> (a sun.nio.ch.Util$2)
> - locked <0x000000051941f820> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005190edc80> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-139-thread-1" prio=10 tid=0x00007f8580002800 nid=0x3f3e runnable
> [0x00007f858c28b000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x000000051913eb50> (a sun.nio.ch.Util$2)
> - locked <0x000000051913eca0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000518c9c4e8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-148-thread-1" prio=10 tid=0x00007f856844c000 nid=0x3f3d waiting on
> condition [0x00007f858c38c000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005190cfe98> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-146-thread-1" prio=10 tid=0x00007f85dc006800 nid=0x3f3c waiting on
> condition [0x00007f858c48d000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for  <0x0000000519018558> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>  at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-147-thread-1" prio=10 tid=0x00007f85e82b1000 nid=0x3f3b waiting on
> condition [0x00007f858c58e000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051903b488> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> "pool-145-thread-1" prio=10 tid=0x00007f8574027000 nid=0x3f3a waiting on
> condition [0x00007f858c68f000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for  <0x000000051903b998> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>  at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-141-thread-1" prio=10 tid=0x00007f85e8159000 nid=0x3f39 runnable
> [0x00007f861cd8e000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005190db9e0> (a sun.nio.ch.Util$2)
> - locked <0x00000005190db998> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000518c9c020> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-138-thread-1" prio=10 tid=0x00007f856844a800 nid=0x3f38 runnable
> [0x00007f861cf90000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x00000005190e1b68> (a sun.nio.ch.Util$2)
> - locked <0x00000005190e1b80> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x0000000518c9c608> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-143-thread-1" prio=10 tid=0x00007f85dc003000 nid=0x3f37 runnable
> [0x00007f861d091000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519018fa8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519018fc0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005190e2228> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "pool-137-thread-1" prio=10 tid=0x00007f8574007000 nid=0x3f36 runnable
> [0x00007f861ce8f000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>  - locked <0x0000000519003dc8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519003de0> (a java.util.Collections$UnmodifiableSet)
>  - locked <0x00000005190ea1a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>  at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  at java.lang.Thread.run(Thread.java:662)
>
> "1407844613@qtp-1263282395-1 - Acceptor0 SocketConnector@0.0.0.0:54321"
> prio=10 tid=0x00007f85e404a800 nid=0x3f35 runnable [0x00007f858c790000]
>    java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
>  - locked <0x00000005190e2720> (a java.net.SocksSocketImpl)
> at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>  at java.net.ServerSocket.accept(ServerSocket.java:430)
> at org.mortbay.jetty.bio.SocketConnector.accept(SocketConnector.java:99)
>  at
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>
> "208924888@qtp-1263282395-0" prio=10 tid=0x00007f85e4021800 nid=0x3f34 in
> Object.wait() [0x00007f861c15b000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
>  - waiting on <0x00000005190d0528> (a
> org.mortbay.thread.QueuedThreadPool$PoolThread)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
>  - locked <0x00000005190d0528> (a
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>
> "lifecycleSupervisor-1-9" prio=10 tid=0x00007f85e401a800 nid=0x3f33
> waiting on condition [0x00007f861c687000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518c85670> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> "lifecycleSupervisor-1-8" prio=10 tid=0x00007f85e4018800 nid=0x3f32
> waiting on condition [0x00007f861c788000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for  <0x0000000518c85670> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>  at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "lifecycleSupervisor-1-7" prio=10 tid=0x00007f85e4016800 nid=0x3f31
> waiting on condition [0x00007f861c889000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518c85670> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> "lifecycleSupervisor-1-6" prio=10 tid=0x00007f85e4014800 nid=0x3f30
> waiting on condition [0x00007f861c98a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for  <0x0000000518c85670> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>  at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "lifecycleSupervisor-1-5" prio=10 tid=0x00007f85e4013000 nid=0x3f2f
> waiting on condition [0x00007f861ca8b000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518c85670> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> "lifecycleSupervisor-1-4" prio=10 tid=0x00007f85e4028000 nid=0x3f2e
> waiting on condition [0x00007f861cb8c000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for  <0x0000000518c85670> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>  at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "lifecycleSupervisor-1-3" prio=10 tid=0x00007f85e4026800 nid=0x3f2d
> waiting on condition [0x00007f861cc8d000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518c85670> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> "Log-BackgroundWorker-ch1" prio=10 tid=0x00007f85dc001000 nid=0x3f27
> waiting on condition [0x00007f861d299000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for  <0x0000000519017718> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>  at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "Log-BackgroundWorker-ch2" prio=10 tid=0x00007f85e80ca000 nid=0x3f26
> waiting on condition [0x00007f861d39a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051903b4e8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> "lifecycleSupervisor-1-2" prio=10 tid=0x00007f85e4025000 nid=0x3f25
> waiting on condition [0x00007f861d6ba000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for  <0x0000000518c85670> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>  at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "lifecycleSupervisor-1-1" prio=10 tid=0x00007f85e400e800 nid=0x3f24
> waiting on condition [0x00007f861d7bb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518c85670> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> "conf-file-poller-0" prio=10 tid=0x00007f85f0001000 nid=0x3f23 waiting on
> condition [0x00007f861d8bc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for  <0x0000000518ec72f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>  at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>  at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>  at java.lang.Thread.run(Thread.java:662)
>
> "DestroyJavaVM" prio=10 tid=0x00007f86a800c800 nid=0x3efa waiting on
> condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "lifecycleSupervisor-1-0" prio=10 tid=0x00007f86a89f5800 nid=0x3f22
> waiting on condition [0x00007f861d9bd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>  at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518c85670> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>  at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> "Low Memory Detector" daemon prio=10 tid=0x00007f86a831d800 nid=0x3f20
> runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread1" daemon prio=10 tid=0x00007f86a831b000 nid=0x3f1f
> waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" daemon prio=10 tid=0x00007f86a8319000 nid=0x3f1e
> waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x00007f86a8316800 nid=0x3f1d
> runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (Concurrent GC)" daemon prio=10
> tid=0x00007f86a8314800 nid=0x3f1c waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x00007f86a82f9800 nid=0x3f1b in
> Object.wait() [0x00007f861e3eb000]
>    java.lang.Thread.State: WAITING (on object monitor)
>  at java.lang.Object.wait(Native Method)
> - waiting on <0x0000000518c843c0> (a java.lang.ref.ReferenceQueue$Lock)
>  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> - locked <0x0000000518c843c0> (a java.lang.ref.ReferenceQueue$Lock)
>  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=0x00007f86a82f7800 nid=0x3f1a in
> Object.wait() [0x00007f861e4ec000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
>  - waiting on <0x0000000518c80860> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:485)
>  at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> - locked <0x0000000518c80860> (a java.lang.ref.Reference$Lock)
>
> "VM Thread" prio=10 tid=0x00007f86a82f1000 nid=0x3f19 runnable
>
> "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f86a801b800
> nid=0x3efb runnable
>
> "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f86a801d000
> nid=0x3efc runnable
>
> "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f86a801f000
> nid=0x3efd runnable
>
> "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f86a8021000
> nid=0x3efe runnable
>
> "Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x00007f86a8022800
> nid=0x3eff runnable
>
> "Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x00007f86a8024800
> nid=0x3f00 runnable
>
> "Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x00007f86a8026000
> nid=0x3f01 runnable
>
> "Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x00007f86a8028000
> nid=0x3f02 runnable
>
> "Gang worker#8 (Parallel GC Threads)" prio=10 tid=0x00007f86a802a000
> nid=0x3f03 runnable
>
> "Gang worker#9 (Parallel GC Threads)" prio=10 tid=0x00007f86a802b800
> nid=0x3f04 runnable
>
> "Gang worker#10 (Parallel GC Threads)" prio=10 tid=0x00007f86a802d800
> nid=0x3f05 runnable
>
> "Gang worker#11 (Parallel GC Threads)" prio=10 tid=0x00007f86a802f800
> nid=0x3f06 runnable
>
> "Gang worker#12 (Parallel GC Threads)" prio=10 tid=0x00007f86a8031000
> nid=0x3f07 runnable
>
> "Gang worker#13 (Parallel GC Threads)" prio=10 tid=0x00007f86a8033000
> nid=0x3f08 runnable
>
> "Gang worker#14 (Parallel GC Threads)" prio=10 tid=0x00007f86a8035000
> nid=0x3f09 runnable
>
> "Gang worker#15 (Parallel GC Threads)" prio=10 tid=0x00007f86a8036800
> nid=0x3f0a runnable
>
> "Gang worker#16 (Parallel GC Threads)" prio=10 tid=0x00007f86a8038800
> nid=0x3f0b runnable
>
> "Gang worker#17 (Parallel GC Threads)" prio=10 tid=0x00007f86a803a000
> nid=0x3f0c runnable
>
> "Gang worker#18 (Parallel GC Threads)" prio=10 tid=0x00007f86a803c000
> nid=0x3f0d runnable
>
> "Gang worker#19 (Parallel GC Threads)" prio=10 tid=0x00007f86a803e000
> nid=0x3f0e runnable
>
> "Gang worker#20 (Parallel GC Threads)" prio=10 tid=0x00007f86a803f800
> nid=0x3f0f runnable
>
> "Gang worker#21 (Parallel GC Threads)" prio=10 tid=0x00007f86a8041800
> nid=0x3f10 runnable
>
> "Gang worker#22 (Parallel GC Threads)" prio=10 tid=0x00007f86a8043800
> nid=0x3f11 runnable
>
> "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f86a82aa000
> nid=0x3f18 runnable
> "Gang worker#0 (Parallel CMS Threads)" prio=10 tid=0x00007f86a829e800
> nid=0x3f12 runnable
>
> "Gang worker#1 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a0000
> nid=0x3f13 runnable
>
> "Gang worker#2 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a2000
> nid=0x3f14 runnable
>
> "Gang worker#3 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a3800
> nid=0x3f15 runnable
>
> "Gang worker#4 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a5800
> nid=0x3f16 runnable
>
> "Gang worker#5 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a7800
> nid=0x3f17 runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x00007f86a8329000 nid=0x3f21
> waiting on condition
>
> JNI global references: 1749
>
>
>
> --
>
>
> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com
>
> Pankaj Gupta | Software Engineer
>
> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>
>
> United States | Canada | United Kingdom | Germany
>
>
> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
> !
>
>
>


-- 


*P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com

Pankaj Gupta | Software Engineer

*BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com


United States | Canada | United Kingdom | Germany


We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
!

Re: Lock contention in FileChannel

[Hari Shreedharan <hs...@cloudera.com>]

Since the channel is designed to make sure that events are not duplicated to multiple sinks, and to protect against corruption due to concurrency issues, we do not need the locking in the channel's flume event queue. It is unlikely that locking is what is causing performance issues because the channel is heavily I/O bound. If you take a series of thread dumps, you will probably see that those threads are moving forward and the ones reading/writing from/to disk are the ones which are slower. These locks are unlikely to hit performance much.

Thanks,
Hari


On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:

> Hi,
> 
> Spent some more time debugging issues with FileChannel. The problem seems to lock contention reading from FlumeEventQueue:
> 
> I see a lot of threads like this: 
> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10 tid=0x00007f857b378800 nid=0x404d waiting for monitor entry [0x00007f821afee000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>         - waiting to lock <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
>         at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>         at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>         at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>         at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>         at org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:662)
> 
> 
> I have two file channels and 8 Avro Sinks per file channel. I added more sinks because they weren't draining fast enough. It seems like they send the batch then wait for an ack before sending again, thus sends are not pipelined and having more sinks seemed like a good way of getting some parallelism. 
> 
> Here's the full stack trace:
> 2013-08-13 15:30:32
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02 mixed mode):
> 
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000] 
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000056d932120> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
> at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
> at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
> at $Proxy7.appendBatch(Unknown Source)
> at org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
> at org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000 nid=0x41a1 waiting on condition [0x0000000000000000] 
>    java.lang.Thread.State: RUNNABLE
> 
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005194c3340> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191a16d8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519174e08> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519174e98> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005195f9670> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519174ce8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191a1918> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191e7f60> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153 runnable [0x00007f8210e4d000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
> - locked <0x000000054a2917a8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519174f58> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146 runnable [0x00007f8210f4e000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
> - locked <0x0000000546e5dd28> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118 runnable [0x00007f821104f000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
> - locked <0x0000000544f763c0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111 runnable [0x00007f8211150000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
> - locked <0x000000054473d5a8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110 runnable [0x00007f8211251000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
> - locked <0x00000005437f4f70> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f runnable [0x00007f8211352000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
> - locked <0x00000005422a9018> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e runnable [0x00007f8211453000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
> - locked <0x0000000541063d50> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d runnable [0x00007f8211554000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
> - locked <0x0000000540dc1110> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a runnable [0x00007f8211655000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
> - locked <0x000000053fbbc3b0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109 runnable [0x00007f8211756000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
> - locked <0x000000053f812860> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd runnable [0x00007f8211857000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
> - locked <0x000000053e6e5570> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519175138> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519174c28> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191751c8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0 runnable [0x00007f8211c5b000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
> - locked <0x0000000536579db0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191750a8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1 runnable [0x00007f8211e5d000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
> - locked <0x00000005313943a8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051936f460> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051936dfa8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de runnable [0x00007f8212160000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
> - locked <0x000000052dad4468> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd runnable [0x00007f8212261000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
> - locked <0x000000052c73c5f0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc runnable [0x00007f8212362000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
> - locked <0x000000052b8da998> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db waiting for monitor entry [0x00007f8212462000] 
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
> - waiting to lock <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
> at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051936df18> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518fe7ae8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8 runnable [0x00007f8212766000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
> - locked <0x00000005263d6ba8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7 runnable [0x00007f8212867000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
> - locked <0x0000000525d1d0e0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6 runnable [0x00007f8212968000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
> - locked <0x0000000525a689d8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5 runnable [0x00007f8212a69000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
> - locked <0x0000000524e1e7d8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4 runnable [0x00007f8212b6a000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
> - locked <0x0000000524d437a8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-36" prio=10 tid=0x00007f85843ad800 nid=0x40d3 runnable [0x00007f8212c6b000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000524cf4140> (a sun.nio.ch.Util$2)
> - locked <0x0000000524cf4128> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000524ab32e8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-35" prio=10 tid=0x00007f85d83a9000 nid=0x40d2 runnable [0x00007f8212d6c000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005244877c8> (a sun.nio.ch.Util$2)
> - locked <0x00000005244877b0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000523f5b660> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-34" prio=10 tid=0x00007f85d838e000 nid=0x40d1 runnable [0x00007f8212e6d000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000052419e490> (a sun.nio.ch.Util$2)
> - locked <0x000000052419e478> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000523f5b6f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-34" prio=10 tid=0x00007f8588390800 nid=0x40d0 waiting for monitor entry [0x00007f8212f6d000] 
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
> - waiting to lock <0x0000000519111590> (a org.apache.flume.channel.file.LogFileV3$Writer)
> at org.apache.flume.channel.file.Log.put(Log.java:507)
> at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
> at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-35" prio=10 tid=0x00007f85843ab800 nid=0x40cf runnable [0x00007f821306f000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000523d40938> (a sun.nio.ch.Util$2)
> - locked <0x0000000523d40aa0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000523ecec18> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-37" prio=10 tid=0x00007f85803e3800 nid=0x40ce waiting for monitor entry [0x00007f821316f000] 
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
> - waiting to lock <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
> at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-34" prio=10 tid=0x00007f8584391000 nid=0x40cd runnable [0x00007f8213271000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000523935540> (a sun.nio.ch.Util$2)
> - locked <0x0000000523935528> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000523788ed8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-33" prio=10 tid=0x00007f85d8373800 nid=0x40cc runnable [0x00007f8213372000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005239ec658> (a sun.nio.ch.Util$2)
> - locked <0x00000005239ec640> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000523845410> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-33" prio=10 tid=0x00007f8588375800 nid=0x40cb waiting for monitor entry [0x00007f8213472000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
> - waiting to lock <0x0000000519111590> (a org.apache.flume.channel.file.LogFileV3$Writer)
> at org.apache.flume.channel.file.Log.put(Log.java:507)
> at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
> at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-33" prio=10 tid=0x00007f8584376000 nid=0x40ca runnable [0x00007f8213574000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005237e3028> (a sun.nio.ch.Util$2)
> - locked <0x00000005237e3010> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000523788e90> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-32" prio=10 tid=0x00007f85d8358800 nid=0x40c9 waiting for monitor entry [0x00007f8213674000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
> - waiting to lock <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
> at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-32" prio=10 tid=0x00007f858835b000 nid=0x40c8 runnable [0x00007f8213776000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000523343688> (a sun.nio.ch.Util$2)
> - locked <0x0000000523343670> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000052305e5c8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-36" prio=10 tid=0x00007f85803c9000 nid=0x40c7 runnable [0x00007f8213877000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005230fa580> (a sun.nio.ch.Util$2)
> - locked <0x00000005230fa568> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000522f5ca50> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-32" prio=10 tid=0x00007f858435b000 nid=0x40c6 runnable [0x00007f8213978000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005231367a8> (a sun.nio.ch.Util$2)
> - locked <0x0000000523136790> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000052305e658> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-31" prio=10 tid=0x00007f85d833e000 nid=0x40c5 runnable [0x00007f8213a79000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000522db30a0> (a sun.nio.ch.Util$2)
> - locked <0x0000000522db3088> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000522bda430> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-31" prio=10 tid=0x00007f8588340000 nid=0x40c4 runnable [0x00007f8213b7a000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000522c16518> (a sun.nio.ch.Util$2)
> - locked <0x0000000522c16500> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000522bd97e8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-30" prio=10 tid=0x00007f85d8323800 nid=0x40c3 runnable [0x00007f8213c7b000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000522bd1a20> (a sun.nio.ch.Util$2)
> - locked <0x0000000522bd1a08> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000522bda3a0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-31" prio=10 tid=0x00007f8584340800 nid=0x40c2 runnable [0x00007f8213d7c000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005228d1608> (a sun.nio.ch.Util$2)
> - locked <0x00000005228d15f0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005228a6aa0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-30" prio=10 tid=0x00007f8588325000 nid=0x40c1 runnable [0x00007f8213e7d000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000052277b088> (a sun.nio.ch.Util$2)
> - locked <0x000000052277b070> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005225ed3b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-35" prio=10 tid=0x00007f85803ae000 nid=0x40c0 runnable [0x00007f8213f7e000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000522737380> (a sun.nio.ch.Util$2)
> - locked <0x0000000522737368> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005225ed328> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-29" prio=10 tid=0x00007f85d8308800 nid=0x40bf runnable [0x00007f821407f000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005226034a8> (a sun.nio.ch.Util$2)
> - locked <0x0000000522603490> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005225ed298> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-28" prio=10 tid=0x00007f85d82ed800 nid=0x40be runnable [0x00007f8214180000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000522359408> (a sun.nio.ch.Util$2)
> - locked <0x00000005223593f0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005221e7278> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-27" prio=10 tid=0x00007f85d82d2800 nid=0x40bd runnable [0x00007f8214281000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005222c0478> (a sun.nio.ch.Util$2)
> - locked <0x00000005222c0460> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005221e6e40> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-29" prio=10 tid=0x00007f858830a800 nid=0x40bc runnable [0x00007f8214382000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000521ddf488> (a sun.nio.ch.Util$2)
> - locked <0x0000000521ddf470> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000521c88d38> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-30" prio=10 tid=0x00007f8584326000 nid=0x40bb waiting for monitor entry [0x00007f8214482000] 
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
> - waiting to lock <0x0000000519111590> (a org.apache.flume.channel.file.LogFileV3$Writer)
> at org.apache.flume.channel.file.Log.put(Log.java:507)
> at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
> at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-26" prio=10 tid=0x00007f85d82b8000 nid=0x40ba runnable [0x00007f8214584000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000521cbcb90> (a sun.nio.ch.Util$2)
> - locked <0x0000000521cbcb78> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000521279250> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-28" prio=10 tid=0x00007f85882ef800 nid=0x40b9 runnable [0x00007f8214685000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000521a1c2d0> (a sun.nio.ch.Util$2)
> - locked <0x0000000521a1c2b8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005218f1508> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-34" prio=10 tid=0x00007f8580393800 nid=0x40b8 runnable [0x00007f8214786000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000052181ce58> (a sun.nio.ch.Util$2)
> - locked <0x000000052181ce40> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005212791c0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-29" prio=10 tid=0x00007f858430b800 nid=0x40b7 runnable [0x00007f8214887000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005219175f0> (a sun.nio.ch.Util$2)
> - locked <0x0000000521746090> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000521278e60> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-27" prio=10 tid=0x00007f85882d4800 nid=0x40b6 runnable [0x00007f8214988000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000052163e9c8> (a sun.nio.ch.Util$2)
> - locked <0x000000052163e9b0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000521279130> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-33" prio=10 tid=0x00007f8580378800 nid=0x40b5 runnable [0x00007f8214a89000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000052152cb38> (a sun.nio.ch.Util$2)
> - locked <0x000000052152cb20> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000521278d40> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-28" prio=10 tid=0x00007f85842f0800 nid=0x40b4 runnable [0x00007f8214b8a000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005213ba3e0> (a sun.nio.ch.Util$2)
> - locked <0x00000005213ba3c8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000521278020> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-27" prio=10 tid=0x00007f85842d5800 nid=0x40b3 runnable [0x00007f8214c8b000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005213ed660> (a sun.nio.ch.Util$2)
> - locked <0x00000005213ed648> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000521278c20> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-26" prio=10 tid=0x00007f85882b9800 nid=0x40b2 runnable [0x00007f8214d8c000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000052133b7c0> (a sun.nio.ch.Util$2)
> - locked <0x000000052133b7a8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005212790a0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-32" prio=10 tid=0x00007f858035d800 nid=0x40af runnable [0x00007f8214e8d000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000520e53be0> (a sun.nio.ch.Util$2)
> - locked <0x0000000520e53bc8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000520f3bb50> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-25" prio=10 tid=0x00007f85d829d000 nid=0x40ae runnable [0x00007f8214f8e000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000520e09a30> (a sun.nio.ch.Util$2)
> - locked <0x0000000520e09a18> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000520f3b6d0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-31" prio=10 tid=0x00007f8580342800 nid=0x40ad runnable [0x00007f821508f000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000520ef70e0> (a sun.nio.ch.Util$2)
> - locked <0x0000000520ef70c8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000520f3b208> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-26" prio=10 tid=0x00007f85842ba800 nid=0x40ac runnable [0x00007f8215190000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000520c9dfc0> (a sun.nio.ch.Util$2)
> - locked <0x0000000520c9dfa8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000520bb3cc0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-25" prio=10 tid=0x00007f85842a0000 nid=0x40ab runnable [0x00007f8215291000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000520b941a8> (a sun.nio.ch.Util$2)
> - locked <0x0000000520b94190> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000520bb30c0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-24" prio=10 tid=0x00007f85d8282800 nid=0x40aa runnable [0x00007f8215392000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000520952288> (a sun.nio.ch.Util$2)
> - locked <0x0000000520952270> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000520356888> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-30" prio=10 tid=0x00007f8580328000 nid=0x40a9 runnable [0x00007f8215493000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000520a2fa20> (a sun.nio.ch.Util$2)
> - locked <0x0000000520a2fa08> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000520983da0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-23" prio=10 tid=0x00007f85d8267800 nid=0x40a8 runnable [0x00007f8215594000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000052095c470> (a sun.nio.ch.Util$2)
> - locked <0x000000052095c458> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000520356840> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-29" prio=10 tid=0x00007f858030d000 nid=0x40a7 runnable [0x00007f8215695000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000052065b178> (a sun.nio.ch.Util$2)
> - locked <0x000000052065b160> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000520357320> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-28" prio=10 tid=0x00007f85802f2800 nid=0x40a6 runnable [0x00007f8215796000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000052043cad0> (a sun.nio.ch.Util$2)
> - locked <0x000000052043cab8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005203567b0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-25" prio=10 tid=0x00007f858829f000 nid=0x40a5 runnable [0x00007f8215897000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000052041a8a8> (a sun.nio.ch.Util$2)
> - locked <0x000000052041a890> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000520356690> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-22" prio=10 tid=0x00007f85d824c800 nid=0x40a4 runnable [0x00007f8215998000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000052036b380> (a sun.nio.ch.Util$2)
> - locked <0x000000052036b368> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000520357290> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-24" prio=10 tid=0x00007f8584285800 nid=0x40a3 runnable [0x00007f8215a99000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000052007ac40> (a sun.nio.ch.Util$2)
> - locked <0x000000052007ac28> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051fe41948> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-27" prio=10 tid=0x00007f85802d8000 nid=0x40a2 runnable [0x00007f8215b9a000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005201a2ae8> (a sun.nio.ch.Util$2)
> - locked <0x00000005201a2ad0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051ffda458> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-21" prio=10 tid=0x00007f85d8232000 nid=0x40a1 waiting for monitor entry [0x00007f8215c9a000] 
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
> - waiting to lock <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
> at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-26" prio=10 tid=0x00007f85802bd800 nid=0x40a0 runnable [0x00007f8215d9c000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051ff20e78> (a sun.nio.ch.Util$2)
> - locked <0x000000051ff20e60> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051fe41900> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-24" prio=10 tid=0x00007f8588284000 nid=0x409f runnable [0x00007f8215e9d000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051fbc1ca0> (a sun.nio.ch.Util$2)
> - locked <0x000000051fbc1c88> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f9a7978> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-25" prio=10 tid=0x00007f85802a2800 nid=0x409e runnable [0x00007f8215f9e000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051fa3b5b8> (a sun.nio.ch.Util$2)
> - locked <0x000000051fa3b5a0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f3155b0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-20" prio=10 tid=0x00007f85d8217800 nid=0x409d runnable [0x00007f821609f000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051f83c838> (a sun.nio.ch.Util$2)
> - locked <0x000000051f83c820> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f3150e8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-23" prio=10 tid=0x00007f8588269800 nid=0x409c runnable [0x00007f82161a0000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051f78b9b0> (a sun.nio.ch.Util$2)
> - locked <0x000000051f78b998> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f3156d0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-22" prio=10 tid=0x00007f858824e800 nid=0x409b runnable [0x00007f82162a1000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051f7c0cc8> (a sun.nio.ch.Util$2)
> - locked <0x000000051f7c0cb0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f3151c0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-23" prio=10 tid=0x00007f858426a800 nid=0x409a runnable [0x00007f82163a2000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051f6065a8> (a sun.nio.ch.Util$2)
> - locked <0x000000051f606590> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f3d9668> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-22" prio=10 tid=0x00007f858424f800 nid=0x4099 runnable [0x00007f82164a3000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051f56d710> (a sun.nio.ch.Util$2)
> - locked <0x000000051f56d6f8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f314f80> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-21" prio=10 tid=0x00007f8588233800 nid=0x4098 runnable [0x00007f82165a4000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051f3bcd68> (a sun.nio.ch.Util$2)
> - locked <0x000000051f3bcd50> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f315400> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-19" prio=10 tid=0x00007f85d81fc800 nid=0x4097 runnable [0x00007f82166a5000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051f53b198> (a sun.nio.ch.Util$2)
> - locked <0x000000051f53b180> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051f315760> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-20" prio=10 tid=0x00007f8588218800 nid=0x4096 waiting for monitor entry [0x00007f82167a5000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
> - waiting to lock <0x0000000519111590> (a org.apache.flume.channel.file.LogFileV3$Writer)
> at org.apache.flume.channel.file.Log.put(Log.java:507)
> at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
> at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-18" prio=10 tid=0x00007f85d81e1800 nid=0x4095 runnable [0x00007f82168a7000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051ee92ba0> (a sun.nio.ch.Util$2)
> - locked <0x000000051ee92b88> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051ed19740> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-19" prio=10 tid=0x00007f85881fd800 nid=0x4094 runnable [0x00007f82169a8000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051ee5da40> (a sun.nio.ch.Util$2)
> - locked <0x000000051ee5da28> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051e77a940> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-17" prio=10 tid=0x00007f85d81c6800 nid=0x4093 runnable [0x00007f8216aa9000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051ecb1730> (a sun.nio.ch.Util$2)
> - locked <0x000000051ecb1718> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051e77a8b0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-21" prio=10 tid=0x00007f8584234800 nid=0x4092 runnable [0x00007f8216baa000]
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051e899660> (a sun.nio.ch.Util$2)
> - locked <0x000000051e899648> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051e77a310> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-24" prio=10 tid=0x00007f8580287000 nid=0x4091 runnable [0x00007f8216caa000]
>    java.lang.Thread.State: RUNNABLE
> at org.apache.flume.channel.file.FlumeEventQueue.addTail(FlumeEventQueue.java:165)
> - locked <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:529)
> - locked <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
> at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-23" prio=10 tid=0x00007f858026c000 nid=0x4090 runnable [0x00007f8216dac000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051ea8b6f0> (a sun.nio.ch.Util$2)
> - locked <0x000000051ea8b6d8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051e77a820> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-16" prio=10 tid=0x00007f85d81ac000 nid=0x408f runnable [0x00007f8216ead000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051e7c8808> (a sun.nio.ch.Util$2)
> - locked <0x000000051e7c87f0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051e77a280> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-20" prio=10 tid=0x00007f8584219800 nid=0x408e runnable [0x00007f8216fae000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051e8fffa8> (a sun.nio.ch.Util$2)
> - locked <0x000000051e8fff90> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051e77a700> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-22" prio=10 tid=0x00007f8580251000 nid=0x408d runnable [0x00007f82170af000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051e4dc398> (a sun.nio.ch.Util$2)
> - locked <0x000000051e4dc380> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051dcbeef8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-19" prio=10 tid=0x00007f85841fe800 nid=0x408c runnable [0x00007f82171b0000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051e221520> (a sun.nio.ch.Util$2)
> - locked <0x000000051e221508> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051e407860> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-18" prio=10 tid=0x00007f85841e3800 nid=0x408b runnable [0x00007f82172b1000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051e465298> (a sun.nio.ch.Util$2)
> - locked <0x000000051e465280> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051dcbef88> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-18" prio=10 tid=0x00007f85881e3000 nid=0x408a runnable [0x00007f82173b2000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051dea12e0> (a sun.nio.ch.Util$2)
> - locked <0x000000051dea12c8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051cfd4d78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-15" prio=10 tid=0x00007f85d8191000 nid=0x4089 runnable [0x00007f82174b3000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051de322b8> (a sun.nio.ch.Util$2)
> - locked <0x000000051de322a0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051dcbec28> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-14" prio=10 tid=0x00007f85d8176800 nid=0x4088 runnable [0x00007f82175b4000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051dea29a8> (a sun.nio.ch.Util$2)
> - locked <0x000000051dea2990> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051dcbf018> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-21" prio=10 tid=0x00007f8580236800 nid=0x4087 runnable [0x00007f82176b5000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051dca95d0> (a sun.nio.ch.Util$2)
> - locked <0x000000051dca95b8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051cfd4e08> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-17" prio=10 tid=0x00007f85841c8800 nid=0x4086 runnable [0x00007f82177b6000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051da8d058> (a sun.nio.ch.Util$2)
> - locked <0x000000051da8d040> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051cd693f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-20" prio=10 tid=0x00007f858021b800 nid=0x4085 runnable [0x00007f82178b7000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051d95a5b0> (a sun.nio.ch.Util$2)
> - locked <0x000000051d95a598> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051cfd4e98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-16" prio=10 tid=0x00007f85841ad800 nid=0x4083 runnable [0x00007f82179b8000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051d92a278> (a sun.nio.ch.Util$2)
> - locked <0x000000051d92a260> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051cfd4fb8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-13" prio=10 tid=0x00007f85d815b800 nid=0x4082 runnable [0x00007f8217ab9000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051d734788> (a sun.nio.ch.Util$2)
> - locked <0x000000051d734770> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051cd69488> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-19" prio=10 tid=0x00007f8580200800 nid=0x4081 runnable [0x00007f8217bba000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051d4268e8> (a sun.nio.ch.Util$2)
> - locked <0x000000051d4268d0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051d139810> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-15" prio=10 tid=0x00007f8584192800 nid=0x4080 runnable [0x00007f8217cbb000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051d4cda88> (a sun.nio.ch.Util$2)
> - locked <0x000000051d4cda70> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051babcf98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-17" prio=10 tid=0x00007f85881c8000 nid=0x407f runnable [0x00007f8217dbc000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051d32ffe8> (a sun.nio.ch.Util$2)
> - locked <0x000000051d32ffd0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051cd691b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-14" prio=10 tid=0x00007f8584177800 nid=0x407e runnable [0x00007f8217ebd000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051d506c60> (a sun.nio.ch.Util$2)
> - locked <0x000000051d506c48> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051cfd5048> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-16" prio=10 tid=0x00007f85881ad000 nid=0x407d runnable [0x00007f8217fbe000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051d1223e0> (a sun.nio.ch.Util$2)
> - locked <0x000000051d1223c8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051c6c7b40> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-15" prio=10 tid=0x00007f8588192800 nid=0x407c runnable [0x00007f82180bf000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051d146f50> (a sun.nio.ch.Util$2)
> - locked <0x000000051d146f38> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051cd69248> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-13" prio=10 tid=0x00007f858415c800 nid=0x407b runnable [0x00007f82181c0000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051d11d1b8> (a sun.nio.ch.Util$2)
> - locked <0x000000051d11d1a0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051c6c7c18> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-12" prio=10 tid=0x00007f85d8141000 nid=0x407a runnable [0x00007f82182c1000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051cd01a78> (a sun.nio.ch.Util$2)
> - locked <0x000000051cd01a60> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051cd695a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-11" prio=10 tid=0x00007f85d8126800 nid=0x4079 runnable [0x00007f82183c2000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051cfa6f88> (a sun.nio.ch.Util$2)
> - locked <0x000000051cfa6f70> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051ba67370> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-14" prio=10 tid=0x00007f8588177800 nid=0x4078 runnable [0x00007f82184c3000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051ce5cdd0> (a sun.nio.ch.Util$2)
> - locked <0x000000051ce5cdb8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051c6c7bd0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-12" prio=10 tid=0x00007f8584142000 nid=0x4077 runnable [0x00007f82185c4000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051cc27cb8> (a sun.nio.ch.Util$2)
> - locked <0x000000051cc27ca0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051babd0b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-18" prio=10 tid=0x00007f85801e6000 nid=0x4076 runnable [0x00007f82186c5000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051cb895c8> (a sun.nio.ch.Util$2)
> - locked <0x000000051cc77fd0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051c6c7cf0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-11" prio=10 tid=0x00007f8584127800 nid=0x4075 runnable [0x00007f82187c6000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051ca540d0> (a sun.nio.ch.Util$2)
> - locked <0x000000051caad368> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051c6c7e10> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-13" prio=10 tid=0x00007f858815c800 nid=0x4074 runnable [0x00007f82188c7000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051c953ab8> (a sun.nio.ch.Util$2)
> - locked <0x000000051ca799a8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051babd028> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-17" prio=10 tid=0x00007f85801cb800 nid=0x4073 runnable [0x00007f82189c8000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051c77c3d8> (a sun.nio.ch.Util$2)
> - locked <0x000000051c8137c8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051c3856f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-10" prio=10 tid=0x00007f858410c800 nid=0x4072 runnable [0x00007f8218ac9000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051c7dd2b0> (a sun.nio.ch.Util$2)
> - locked <0x000000051c613988> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051babcde8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-16" prio=10 tid=0x00007f85801b1000 nid=0x4071 runnable [0x00007f8218bca000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051c776580> (a sun.nio.ch.Util$2)
> - locked <0x000000051c6633c8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051c6c7ea0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-9" prio=10 tid=0x00007f85840f1800 nid=0x4070 runnable [0x00007f8218ccb000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051c36b550> (a sun.nio.ch.Util$2)
> - locked <0x000000051c36b568> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051ba675f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-8" prio=10 tid=0x00007f85840d6800 nid=0x406f runnable [0x00007f8218dcc000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051c546b00> (a sun.nio.ch.Util$2)
> - locked <0x000000051c40a178> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051ba67760> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-7" prio=10 tid=0x00007f85840bb800 nid=0x406e runnable [0x00007f8218ecd000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051c2ecf90> (a sun.nio.ch.Util$2)
> - locked <0x000000051c29a100> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051ba675b0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-15" prio=10 tid=0x00007f8580196000 nid=0x406d runnable [0x00007f8218fce000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051c2ee670> (a sun.nio.ch.Util$2)
> - locked <0x000000051c2ee688> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051ba67718> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-12" prio=10 tid=0x00007f8588141800 nid=0x406c runnable [0x00007f82190cf000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051c0bd978> (a sun.nio.ch.Util$2)
> - locked <0x000000051c16cc00> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051babcf08> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-6" prio=10 tid=0x00007f85840a0800 nid=0x406b runnable [0x00007f82191d0000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051c068508> (a sun.nio.ch.Util$2)
> - locked <0x000000051bf825f0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051ba676d0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-11" prio=10 tid=0x00007f8588127000 nid=0x406a runnable [0x00007f82192d1000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051becaa68> (a sun.nio.ch.Util$2)
> - locked <0x000000051c03ead0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051babce78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-10" prio=10 tid=0x00007f858810c800 nid=0x4069 runnable [0x00007f82193d2000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051be31b98> (a sun.nio.ch.Util$2)
> - locked <0x000000051bd56790> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051ba67490> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-14" prio=10 tid=0x00007f858017b000 nid=0x4068 runnable [0x00007f82194d3000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051bccb8c0> (a sun.nio.ch.Util$2)
> - locked <0x000000051bb14420> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051babcd58> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-5" prio=10 tid=0x00007f8584085800 nid=0x4067 runnable [0x00007f82195d4000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051bd004a8> (a sun.nio.ch.Util$2)
> - locked <0x000000051bd49480> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051ba67400> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-4" prio=10 tid=0x00007f858406b000 nid=0x4066 runnable [0x00007f82196d5000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051ba1e928> (a sun.nio.ch.Util$2)
> - locked <0x000000051bccdd00> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051ba66ef0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-9" prio=10 tid=0x00007f85880f1800 nid=0x4065 waiting for monitor entry [0x00007f82197d5000] 
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:202)
> - waiting to lock <0x0000000519111590> (a org.apache.flume.channel.file.LogFileV3$Writer)
> at org.apache.flume.channel.file.Log.put(Log.java:507)
> at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:457)
> at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
> at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
> at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-8" prio=10 tid=0x00007f85880d6800 nid=0x4064 runnable [0x00007f82198d7000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051b5bcca8> (a sun.nio.ch.Util$2)
> - locked <0x000000051b7ef8f0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051b5c7cb8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-7" prio=10 tid=0x00007f85880bb800 nid=0x4063 runnable [0x00007f82199d8000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051b788ce8> (a sun.nio.ch.Util$2)
> - locked <0x000000051b788d00> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051b285260> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-13" prio=10 tid=0x00007f8580160800 nid=0x4062 runnable [0x00007f8219ad9000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051b71a0f8> (a sun.nio.ch.Util$2)
> - locked <0x000000051b750e98> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051aed5a70> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-10" prio=10 tid=0x00007f85d810b800 nid=0x4061 runnable [0x00007f8219bda000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051b517878> (a sun.nio.ch.Util$2)
> - locked <0x000000051b5bab30> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051aed5b90> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-9" prio=10 tid=0x00007f85d80f0800 nid=0x4060 runnable [0x00007f8219cdb000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051b5656e0> (a sun.nio.ch.Util$2)
> - locked <0x000000051b5657a0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051b285338> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-8" prio=10 tid=0x00007f85d80d5800 nid=0x405f runnable [0x00007f8219ddc000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051b3df220> (a sun.nio.ch.Util$2)
> - locked <0x000000051b475118> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051aed5c20> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-6" prio=10 tid=0x00007f85880a0800 nid=0x405e runnable [0x00007f8219edd000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051b11c010> (a sun.nio.ch.Util$2)
> - locked <0x000000051b1eab78> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051af91478> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-7" prio=10 tid=0x00007f85d80bb000 nid=0x405d runnable [0x00007f8219fde000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051b1d36e0> (a sun.nio.ch.Util$2)
> - locked <0x000000051b0328c0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051aed5cb0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-12" prio=10 tid=0x00007f8580145800 nid=0x405c runnable [0x00007f821a0df000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051aeb3070> (a sun.nio.ch.Util$2)
> - locked <0x000000051aeb3088> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051a98a820> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-6" prio=10 tid=0x00007f85d80a0000 nid=0x405b runnable [0x00007f821a1e0000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051aefa920> (a sun.nio.ch.Util$2)
> - locked <0x000000051af3ee40> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051ac28810> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-5" prio=10 tid=0x00007f8588086000 nid=0x405a runnable [0x00007f821a2e1000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051aefe1b0> (a sun.nio.ch.Util$2)
> - locked <0x000000051ad2b858> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051aed5e18> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-11" prio=10 tid=0x00007f858012a800 nid=0x4059 runnable [0x00007f821a3e2000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051ac42588> (a sun.nio.ch.Util$2)
> - locked <0x000000051ad05d70> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051a98a7d8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-3" prio=10 tid=0x00007f8584050000 nid=0x4058 runnable [0x00007f821a4e3000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051ad06100> (a sun.nio.ch.Util$2)
> - locked <0x000000051ad47428> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051ac289c0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-10" prio=10 tid=0x00007f858010f800 nid=0x4057 runnable [0x00007f821a5e4000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051aab5d80> (a sun.nio.ch.Util$2)
> - locked <0x000000051ab8ce38> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051a98a748> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-5" prio=10 tid=0x00007f85d8085800 nid=0x4056 runnable [0x00007f821a6e5000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051a962980> (a sun.nio.ch.Util$2)
> - locked <0x000000051a775fb0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051a842098> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-2" prio=10 tid=0x00007f8584035000 nid=0x4055 runnable [0x00007f821a7e6000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051a817210> (a sun.nio.ch.Util$2)
> - locked <0x000000051a7d2b48> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051a841ca8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-4" prio=10 tid=0x00007f85d806b000 nid=0x4054 runnable [0x00007f821a8e7000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051a903340> (a sun.nio.ch.Util$2)
> - locked <0x000000051a903358> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051a42db50> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-9" prio=10 tid=0x00007f85800f4800 nid=0x4053 runnable [0x00007f821a9e8000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051a4f17d8> (a sun.nio.ch.Util$2)
> - locked <0x000000051a37ae08> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051a22e340> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-4" prio=10 tid=0x00007f858806b000 nid=0x4052 runnable [0x00007f821aae9000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051a4d81e8> (a sun.nio.ch.Util$2)
> - locked <0x000000051a616bb0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051a1dfb40> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-3" prio=10 tid=0x00007f85d8050000 nid=0x4051 runnable [0x00007f821abea000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051a4fb240> (a sun.nio.ch.Util$2)
> - locked <0x000000051a3913b8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051a213878> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-2" prio=10 tid=0x00007f85d8035800 nid=0x4050 runnable [0x00007f821aceb000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051a506248> (a sun.nio.ch.Util$2)
> - locked <0x000000051a506308> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051a42dd48> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-8" prio=10 tid=0x00007f85800d9800 nid=0x404f runnable [0x00007f821adec000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051a2d5170> (a sun.nio.ch.Util$2)
> - locked <0x000000051a2d5188> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051a22e2b0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-7" prio=10 tid=0x00007f85800be800 nid=0x404e runnable [0x00007f821aeed000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051a2d3cb8> (a sun.nio.ch.Util$2)
> - locked <0x000000051a2b8310> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051a213758> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10 tid=0x00007f857b378800 nid=0x404d waiting for monitor entry [0x00007f821afee000] 
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
> - waiting to lock <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
> at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
> at org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f836c01a000 nid=0x404c runnable [0x00007f821b0ef000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051a045bb0> (a sun.nio.ch.Util$2)
> - locked <0x000000051a045bc8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051a000c98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b376800 nid=0x404b waiting on condition [0x00007f821b1f0000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051a045c10> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10 tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.NativeThread.current(Native Method)
> at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
> - locked <0x00000005190ec998> (a java.lang.Object)
> at org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
> at org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
> - locked <0x0000000519111590> (a org.apache.flume.channel.file.LogFileV3$Writer)
> at org.apache.flume.channel.file.Log.take(Log.java:550)
> at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
> at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
> at org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8354035000 nid=0x4049 runnable [0x00007f821b3f2000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051a045c28> (a sun.nio.ch.Util$2)
> - locked <0x000000051a045c40> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051a000ce0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834801a000 nid=0x4048 runnable [0x00007f821b4f3000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051a17bd60> (a sun.nio.ch.Util$2)
> - locked <0x000000051a17bd78> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051a180230> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b374800 nid=0x4047 waiting on condition [0x00007f821b5f4000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051a08ba88> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857338a800 nid=0x4046 waiting on condition [0x00007f821b6f5000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051a147940> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10 tid=0x00007f85f3bff800 nid=0x4045 waiting on condition [0x00007f821b7f6000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000056dca65c0> (a java.util.concurrent.FutureTask$Sync)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:227)
> at java.util.concurrent.FutureTask.get(FutureTask.java:91)
> at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:323)
> at org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:269)
> at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:352)
> at org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8344035000 nid=0x4044 runnable [0x00007f821b8f7000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051a184a90> (a sun.nio.ch.Util$2)
> - locked <0x000000051a184aa8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051a0bb2e0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bfd800 nid=0x4043 waiting on condition [0x00007f821b9f8000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051a147958> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f833001a000 nid=0x4042 runnable [0x00007f821baf9000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051a188f28> (a sun.nio.ch.Util$2)
> - locked <0x000000051a188f40> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519d90758> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8334035000 nid=0x4041 runnable [0x00007f821bbfa000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051a045cd0> (a sun.nio.ch.Util$2)
> - locked <0x000000051a045ce8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051a000d28> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857b372800 nid=0x4040 waiting on condition [0x00007f821bcfb000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f74590> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573388800 nid=0x403f waiting on condition [0x00007f821bdfc000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fb0c90> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832c035000 nid=0x403e runnable [0x00007f821befd000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051a045d30> (a sun.nio.ch.Util$2)
> - locked <0x000000051a045d48> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051a000d70> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bfb800 nid=0x403d waiting on condition [0x00007f821bffe000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519e5d388> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f831801a000 nid=0x403c runnable [0x00007f82341c0000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519ebdb70> (a sun.nio.ch.Util$2)
> - locked <0x0000000519ebdb88> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519d3a778> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f831001a000 nid=0x403b runnable [0x00007f82342c1000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519f47f90> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f47fa8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519e11ca8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaf0000 nid=0x403a waiting on condition [0x00007f82343c2000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f73018> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8308035000 nid=0x4039 runnable [0x00007f82344c3000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519f745a8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f745c0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519ca9f78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573386800 nid=0x4038 waiting on condition [0x00007f82345c4000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fa5ce0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3bf9800 nid=0x4037 waiting on condition [0x00007f82346c5000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fb0ca8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8300035000 nid=0x4036 runnable [0x00007f82347c6000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519f22d20> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f22d38> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519d9f4b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaee000 nid=0x4035 waiting on condition [0x00007f82348c7000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fb0cc0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82fc01a000 nid=0x4034 runnable [0x00007f82349c8000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519f74608> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f74620> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519ca9fc0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82f001a000 nid=0x4033 runnable [0x00007f8234ac9000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519fb0cd8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fb0cf0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519cadd40> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f44800 nid=0x4032 waiting on condition [0x00007f8234bca000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519ebdbd0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3377000 nid=0x4031 waiting on condition [0x00007f8234ccb000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f701c0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82e8035000 nid=0x4030 runnable [0x00007f8234dcc000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519f47ff0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f48008> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519e11cf0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaec000 nid=0x402f waiting on condition [0x00007f8234ecd000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fa5cf8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10 tid=0x00007f857fc38000 nid=0x402e waiting for monitor entry [0x00007f8234fce000] 
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
> - waiting to lock <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
> at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
> at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
> at org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82dc01a000 nid=0x402d runnable [0x00007f82350cf000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519ebdbe8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519ebdc00> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519d3a7c0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82d001a000 nid=0x402c runnable [0x00007f82351d0000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519fb0d38> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fb0d50> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519cadd88> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-144-thread-1" prio=10 tid=0x00007f85d8001000 nid=0x402b runnable [0x00007f82352d1000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519f481a0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f49040> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519e11d80> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6d3800 nid=0x402a waiting on condition [0x00007f82353d2000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f22d80> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f42800 nid=0x4029 waiting on condition [0x00007f82354d3000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fa5d10> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82cc035000 nid=0x4028 runnable [0x00007f82355d4000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519fb0d98> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fb0db0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519caddd0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-3" prio=10 tid=0x00007f8588001000 nid=0x4027 runnable [0x00007f82356d5000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519ebdc48> (a sun.nio.ch.Util$2)
> - locked <0x0000000519ebdc60> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519d3a808> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3375000 nid=0x4026 waiting on condition [0x00007f82357d6000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fa5d28> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82c4035000 nid=0x4025 runnable [0x00007f82358d7000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519f22d98> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f22db0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519d9f500> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857aaea000 nid=0x4024 waiting on condition [0x00007f82359d8000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fa5d40> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a8035000 nid=0x4023 runnable [0x00007f8235ad9000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519fb0df8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fb0e10> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519cade18> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82b401a000 nid=0x4022 runnable [0x00007f8235bda000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519f73030> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f73048> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519e69de8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82ac035000 nid=0x4021 runnable [0x00007f8235cdb000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519f61380> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f61398> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005197600f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6d1800 nid=0x4020 waiting on condition [0x00007f8235ddc000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f22df8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f40800 nid=0x401f waiting on condition [0x00007f8235edd000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fa5d58> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3373000 nid=0x401e waiting on condition [0x00007f8235fde000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519ebdca8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a4035000 nid=0x401d runnable [0x00007f82360df000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519fb0e58> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fb0e70> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519cade60> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a8000 nid=0x401c waiting on condition [0x00007f82361e0000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f22e10> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-6" prio=10 tid=0x00007f85800a3800 nid=0x401b runnable [0x00007f82362e1000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519ebde28> (a sun.nio.ch.Util$2)
> - locked <0x0000000519de7350> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519d3a898> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828801a000 nid=0x401a runnable [0x00007f82363e2000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519fa5d70> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fa5d88> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519d36338> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828c035000 nid=0x4019 runnable [0x00007f82364e3000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519f73090> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f730a8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519e69e30> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8280035000 nid=0x4018 runnable [0x00007f82365e4000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519fb0eb8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519e1a7f8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519cadea8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6cf800 nid=0x4017 waiting on condition [0x00007f82366e5000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f22e28> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572f3e800 nid=0x4016 waiting on condition [0x00007f82367e6000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519f701d8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f3371000 nid=0x4015 waiting on condition [0x00007f82368e7000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519fb0f00> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8278035000 nid=0x4014 runnable [0x00007f82369e8000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519e93628> (a sun.nio.ch.Util$2)
> - locked <0x0000000519e93640> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519d9f548> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a6000 nid=0x4013 waiting on condition [0x00007f8236ae9000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005199bdaf0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f827001a000 nid=0x4012 runnable [0x00007f8236bea000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519fb0f18> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fb0f30> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519cadef0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e6cd800 nid=0x4011 waiting on condition [0x00007f8236ceb000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c47f38> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826c035000 nid=0x4010 runnable [0x00007f8236dec000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519fa5dd0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519fa5de8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519d36380> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826001a000 nid=0x400f runnable [0x00007f8236eed000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519c8c170> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c8c188> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519a73d78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572aeb000 nid=0x400e waiting on condition [0x00007f8236fee000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c7e570> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2e800 nid=0x400d waiting on condition [0x00007f82370ef000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c47f50> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8258035000 nid=0x400b runnable [0x00007f82371f0000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519c4b190> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c4b1a8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005199eb068> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a4000 nid=0x400a waiting on condition [0x00007f82372f1000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c47f68> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f825001a000 nid=0x4009 runnable [0x00007f82373f2000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519c33550> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c33568> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051996b7b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e29c000 nid=0x4008 waiting on condition [0x00007f82374f3000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c47f80> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f824c03f800 nid=0x4007 runnable [0x00007f82375f4000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519c7e588> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e5a0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005199a1910> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8240035000 nid=0x4006 runnable [0x00007f82376f5000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519c47f98> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c47fb0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005199536b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae9000 nid=0x4005 waiting on condition [0x00007f82377f6000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c368c8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2c800 nid=0x4004 waiting on condition [0x00007f82378f7000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c29bf0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8238035000 nid=0x4003 runnable [0x00007f82379f8000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519bd8198> (a sun.nio.ch.Util$2)
> - locked <0x0000000519bd81b0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519b59da0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a6a2000 nid=0x4002 waiting on condition [0x00007f8237af9000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c7e5e8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f8230035000 nid=0x4001 runnable [0x00007f8237bfa000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519c47ff8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c48010> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519953700> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e29a000 nid=0x4000 waiting on condition [0x00007f8237cfb000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c7e600> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f823001a000 nid=0x3fff runnable [0x00007f8237dfc000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519c7e618> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e630> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005199a1958> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f823c01a000 nid=0x3ffe runnable [0x00007f8237efd000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519c48058> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c48070> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519a6d798> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f823801a000 nid=0x3ffd runnable [0x00007f8237ffe000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519c335b0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c335c8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051996b800> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae7000 nid=0x3ffc waiting on condition [0x00007f84001c0000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c480b8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f2a800 nid=0x3ffb waiting on condition [0x00007f84002c1000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519bd81f8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24e800 nid=0x3ffa waiting on condition [0x00007f84003c2000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c29c08> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f824001a000 nid=0x3ff9 runnable [0x00007f84004c3000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519c7e678> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e690> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005199a19a0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e298000 nid=0x3ff8 waiting on condition [0x00007f84005c4000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c67408> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-5" prio=10 tid=0x00007f8580088800 nid=0x3ff7 runnable [0x00007f84006c5000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519c48238> (a sun.nio.ch.Util$2)
> - locked <0x0000000519d5e0d0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519a6d828> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f825c01a000 nid=0x3ff6 runnable [0x00007f84007c6000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519c368e0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c368f8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519953088> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f825801a000 nid=0x3ff5 runnable [0x00007f84008c7000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519c33610> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c33628> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051996b848> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826401a000 nid=0x3ff4 runnable [0x00007f84009c8000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519c7e6d8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e6f0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005199a19e8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572ae5000 nid=0x3ff3 waiting on condition [0x00007f8400ac9000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c730b8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24c800 nid=0x3ff2 waiting on condition [0x00007f8400bca000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c29c20> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2f29000 nid=0x3ff1 waiting on condition [0x00007f8400ccb000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c4b1f0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f826c01a000 nid=0x3ff0 runnable [0x00007f8400dcc000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519bd8210> (a sun.nio.ch.Util$2)
> - locked <0x0000000519bd8228> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519b59de8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857e296800 nid=0x3fef waiting on condition [0x00007f8400ecd000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c8c1d0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f827801a000 nid=0x3fee runnable [0x00007f8400fce000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519c7e738> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e750> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005199a1a30> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828401a000 nid=0x3fed runnable [0x00007f84010cf000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519c33670> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c33688> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051996b890> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828001a000 nid=0x3fec runnable [0x00007f84011d0000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519c29c38> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c29c50> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519955c68> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f828c01a000 nid=0x3feb runnable [0x00007f84012d1000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519c7e798> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e7b0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005199533a0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae6000 nid=0x3fea waiting on condition [0x00007f84013d2000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519bd8270> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a24a800 nid=0x3fe9 waiting on condition [0x00007f84014d3000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c36940> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572273800 nid=0x3fe8 waiting on condition [0x00007f84015d4000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c8c1e8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de54800 nid=0x3fe7 waiting on condition [0x00007f84016d5000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519c67420> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f829001a000 nid=0x3fe6 runnable [0x00007f84017d6000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519c7e7f8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e810> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005199533e8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a401a000 nid=0x3fe5 runnable [0x00007f84018d7000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519c336d0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c336e8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051996b8d8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae4000 nid=0x3fe4 waiting on condition [0x00007f84019d8000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519946350> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-4" prio=10 tid=0x00007f858006d800 nid=0x3fe3 runnable [0x00007f8401ad9000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519bd83d8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519f042a0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519b59e78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82ac01a000 nid=0x3fe2 runnable [0x00007f8401bda000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519c7e858> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c7e870> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519953430> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857a248800 nid=0x3fe1 waiting on condition [0x00007f8401cdb000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519890ec8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82a801a000 nid=0x3fe0 runnable [0x00007f8401ddc000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519c8c200> (a sun.nio.ch.Util$2)
> - locked <0x0000000519c8c218> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519a73dc0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8572271800 nid=0x3fdf waiting on condition [0x00007f8401edd000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198a3170> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de52800 nid=0x3fde waiting on condition [0x00007f8401fde000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198b7610> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82b801a000 nid=0x3fdd runnable [0x00007f84020df000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519947f70> (a sun.nio.ch.Util$2)
> - locked <0x0000000519947f88> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005197bc938> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82c401a000 nid=0x3fdc runnable [0x00007f84021e0000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519946368> (a sun.nio.ch.Util$2)
> - locked <0x0000000519946380> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051983b9b8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82cc01a000 nid=0x3fdb runnable [0x00007f84022e1000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005198a3188> (a sun.nio.ch.Util$2)
> - locked <0x00000005198a31a0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005197300e0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae2000 nid=0x3fda waiting on condition [0x00007f84023e2000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005199463c8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857226f800 nid=0x3fd9 waiting on condition [0x00007f84024e3000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519792380> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82c801a000 nid=0x3fd8 runnable [0x00007f84025e4000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519947fd0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519947fe8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005197bc980> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579597000 nid=0x3fd7 waiting on condition [0x00007f84026e5000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519488cc0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de50800 nid=0x3fd6 waiting on condition [0x00007f84027e6000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198cc640> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82d801a000 nid=0x3fd5 runnable [0x00007f84028e7000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005199463e0> (a sun.nio.ch.Util$2)
> - locked <0x00000005199463f8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051983ba00> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82e401a000 nid=0x3fd4 runnable [0x00007f84029e8000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005198f7fc8> (a sun.nio.ch.Util$2)
> - locked <0x00000005198f7fe0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005194d0160> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82ec01a000 nid=0x3fd3 runnable [0x00007f8402ae9000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005198a31e8> (a sun.nio.ch.Util$2)
> - locked <0x00000005198a3200> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519730128> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f2ae0800 nid=0x3fd2 waiting on condition [0x00007f8402bea000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519948030> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857226d800 nid=0x3fd1 waiting on condition [0x00007f8402ceb000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519946440> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f82e801a000 nid=0x3fd0 runnable [0x00007f8402dec000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519890ee0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519890ef8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005196f56f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579595000 nid=0x3fcf waiting on condition [0x00007f8402eed000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198b7628> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857de4e800 nid=0x3fce waiting on condition [0x00007f8402fee000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198a3248> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830401a000 nid=0x3fcd runnable [0x00007f84030ef000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519946458> (a sun.nio.ch.Util$2)
> - locked <0x0000000519946470> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051983ba48> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830001a000 nid=0x3fcc runnable [0x00007f84031f0000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519937b88> (a sun.nio.ch.Util$2)
> - locked <0x0000000519937ba0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519887cb8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830c01a000 nid=0x3fcb runnable [0x00007f84032f1000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519948048> (a sun.nio.ch.Util$2)
> - locked <0x0000000519948060> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005197bc9c8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f830801a000 nid=0x3fca runnable [0x00007f84033f2000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005199464b8> (a sun.nio.ch.Util$2)
> - locked <0x00000005199464d0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051983ba90> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e2b000 nid=0x3fc9 waiting on condition [0x00007f84034f3000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519943980> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1595000 nid=0x3fc8 waiting on condition [0x00007f84035f4000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198a3260> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857bff5800 nid=0x3fc7 waiting on condition [0x00007f84036f5000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519901138> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da0c000 nid=0x3fc6 waiting on condition [0x00007f84037f6000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005199480a8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832001a000 nid=0x3fc5 runnable [0x00007f84038f7000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519946518> (a sun.nio.ch.Util$2)
> - locked <0x0000000519946530> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051983bad8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832c01a000 nid=0x3fc4 runnable [0x00007f84039f8000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005198a3278> (a sun.nio.ch.Util$2)
> - locked <0x00000005198a3290> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519730170> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f832801a000 nid=0x3fc3 runnable [0x00007f8403af9000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005198f8028> (a sun.nio.ch.Util$2)
> - locked <0x00000005198f8040> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005194d01a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f833401a000 nid=0x3fc2 runnable [0x00007f8403bfa000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005198b7640> (a sun.nio.ch.Util$2)
> - locked <0x00000005198b7658> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005196f3480> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e29000 nid=0x3fc1 waiting on condition [0x00007f8403cfb000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519946578> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1593000 nid=0x3fc0 waiting on condition [0x00007f8403dfc000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005199480c0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579592000 nid=0x3fbf waiting on condition [0x00007f8403efd000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519943998> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da0a800 nid=0x3fbe waiting on condition [0x00007f84401c0000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519946590> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834401a000 nid=0x3fbd runnable [0x00007f84402c1000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519903410> (a sun.nio.ch.Util$2)
> - locked <0x0000000519903428> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005191a1498> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834001a000 nid=0x3fbc runnable [0x00007f84403c2000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519943ef0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519943f08> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005196f5740> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f834c01a000 nid=0x3fbb runnable [0x00007f84404c3000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005199497b0> (a sun.nio.ch.Util$2)
> - locked <0x00000005199497c8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519685f60> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8573ff1800 nid=0x3fba waiting on condition [0x00007f84405c4000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198f8088> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f1591800 nid=0x3fb9 waiting on condition [0x00007f84406c5000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198a32d8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579591000 nid=0x3fb8 waiting on condition [0x00007f84407c6000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005199480d8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f835401a000 nid=0x3fb7 runnable [0x00007f84408c7000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005199465a8> (a sun.nio.ch.Util$2)
> - locked <0x00000005199465c0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051983bb20> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da08800 nid=0x3fb6 waiting on condition [0x00007f84409c8000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005199439b0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-142-thread-1" prio=10 tid=0x00007f8584001000 nid=0x3fb5 runnable [0x00007f8440ac9000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005198b77f0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519a00960> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519952e48> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f837401a000 nid=0x3fb4 runnable [0x00007f8440bca000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519937be8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519937c00> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519887d00> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f836801a000 nid=0x3fb3 runnable [0x00007f8440ccb000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519943f50> (a sun.nio.ch.Util$2)
> - locked <0x0000000519943f68> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005196f5788> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f837001a000 nid=0x3fb2 runnable [0x00007f8440dcc000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519901150> (a sun.nio.ch.Util$2)
> - locked <0x0000000519901168> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519106100> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f158f800 nid=0x3fb1 waiting on condition [0x00007f8440ecd000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519946608> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571e26800 nid=0x3fb0 waiting on condition [0x00007f8440fce000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005198a32f0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913e000 nid=0x3faf waiting on condition [0x00007f84410cf000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005199439c8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f837801a000 nid=0x3fae runnable [0x00007f84411d0000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005199480f0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519948108> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005197bca10> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857da07000 nid=0x3fad waiting on condition [0x00007f84412d1000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519943fb0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f838801a000 nid=0x3fac runnable [0x00007f84413d2000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005198f80a0> (a sun.nio.ch.Util$2)
> - locked <0x00000005198f80b8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005194d01f0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f839401a000 nid=0x3fab runnable [0x00007f84414d3000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519946620> (a sun.nio.ch.Util$2)
> - locked <0x0000000519946638> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051983bb68> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f839001a000 nid=0x3faa runnable [0x00007f84415d4000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519937c48> (a sun.nio.ch.Util$2)
> - locked <0x0000000519937c60> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519887d48> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d0d000 nid=0x3fa9 waiting on condition [0x00007f84416d5000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519654310> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913c000 nid=0x3fa8 waiting on condition [0x00007f84417d6000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005191b54d8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571164000 nid=0x3fa7 waiting on condition [0x00007f84418d7000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519637a20> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83a801a000 nid=0x3fa6 runnable [0x00007f84419d8000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005195f76c8> (a sun.nio.ch.Util$2)
> - locked <0x00000005196430c0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005194ecaf0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83b401a000 nid=0x3fa5 runnable [0x00007f8441ad9000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051965f960> (a sun.nio.ch.Util$2)
> - locked <0x000000051965f978> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519104648> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83b001a000 nid=0x3fa4 runnable [0x00007f8441bda000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519655e28> (a sun.nio.ch.Util$2)
> - locked <0x0000000519655e40> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519104ba0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-3" prio=10 tid=0x00007f8580052800 nid=0x3fa3 runnable [0x00007f8441cdb000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519637b10> (a sun.nio.ch.Util$2)
> - locked <0x0000000519694210> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051916e518> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83bc01a000 nid=0x3fa2 runnable [0x00007f8441ddc000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519597678> (a sun.nio.ch.Util$2)
> - locked <0x0000000519597690> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519106388> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c4800 nid=0x3fa1 waiting on condition [0x00007f8441edd000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519654328> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d0b000 nid=0x3fa0 waiting on condition [0x00007f8441fde000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519676ad0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571162000 nid=0x3f9f waiting on condition [0x00007f84420df000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519655618> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857913a000 nid=0x3f9e waiting on condition [0x00007f84421e0000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005194ae248> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-2" prio=10 tid=0x00007f8580001000 nid=0x3f9d runnable [0x00007f84422e1000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519654340> (a sun.nio.ch.Util$2)
> - locked <0x0000000519654358> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519105380> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83cc01a000 nid=0x3f9c runnable [0x00007f84423e2000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519655e88> (a sun.nio.ch.Util$2)
> - locked <0x0000000519655ea0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519104be8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83d001a000 nid=0x3f9b runnable [0x00007f84424e3000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005194e1350> (a sun.nio.ch.Util$2)
> - locked <0x00000005194e1368> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005191041c8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c3000 nid=0x3f9a waiting on condition [0x00007f84425e4000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051967caf8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83d801a000 nid=0x3f99 runnable [0x00007f84426e5000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519655ee8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519655f00> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519104c30> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83dc01a000 nid=0x3f98 runnable [0x00007f84427e6000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005196543a0> (a sun.nio.ch.Util$2)
> - locked <0x00000005196543b8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005191053c8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d09800 nid=0x3f97 waiting on condition [0x00007f84428e7000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519655630> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8579138000 nid=0x3f96 waiting on condition [0x00007f84429e8000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005196592c8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8571160000 nid=0x3f95 waiting on condition [0x00007f8442ae9000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519655f48> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f840801a000 nid=0x3f94 runnable [0x00007f8442bea000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519655648> (a sun.nio.ch.Util$2)
> - locked <0x0000000519655660> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051914b290> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83e801a000 nid=0x3f93 runnable [0x00007f8442ceb000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519643108> (a sun.nio.ch.Util$2)
> - locked <0x0000000519643120> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005194ecb38> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0d07800 nid=0x3f92 waiting on condition [0x00007f8442dec000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519658428> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83f801a000 nid=0x3f91 runnable [0x00007f8442eed000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005194c9908> (a sun.nio.ch.Util$2)
> - locked <0x00000005194c9920> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519511968> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f83fc01a000 nid=0x3f90 runnable [0x00007f8442fee000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519643168> (a sun.nio.ch.Util$2)
> - locked <0x0000000519643180> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005194ecb80> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578465000 nid=0x3f8f waiting on condition [0x00007f8403ffe000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519654400> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857115e800 nid=0x3f8e waiting on condition [0x00007f84430ef000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005196556a8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5c1000 nid=0x3f8d waiting on condition [0x00007f84431f0000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519654418> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f840c01a000 nid=0x3f8c runnable [0x00007f84432f1000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005196431c8> (a sun.nio.ch.Util$2)
> - locked <0x00000005196431e0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005194ecbc8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f841801a000 nid=0x3f8b runnable [0x00007f84433f2000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519655f60> (a sun.nio.ch.Util$2)
> - locked <0x0000000519655f78> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519104c78> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b4000 nid=0x3f8a waiting on condition [0x00007f84434f3000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005196721e8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5bf000 nid=0x3f89 waiting on condition [0x00007f84435f4000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519654430> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f841c01a000 nid=0x3f88 runnable [0x00007f84436f5000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005196556c0> (a sun.nio.ch.Util$2)
> - locked <0x00000005196556d8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051914b2d8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f842801a000 nid=0x3f87 runnable [0x00007f84437f6000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519643228> (a sun.nio.ch.Util$2)
> - locked <0x0000000519643240> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005194ecc10> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b2000 nid=0x3f86 waiting on condition [0x00007f84438f7000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519654448> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d5bd800 nid=0x3f85 waiting on condition [0x00007f84439f8000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051967cb10> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f843001a000 nid=0x3f84 runnable [0x00007f8443af9000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519655fc0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519655fd8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519104cc0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08b0800 nid=0x3f83 waiting on condition [0x00007f8443bfa000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005196592e0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f843801a000 nid=0x3f82 runnable [0x00007f8443cfb000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519643288> (a sun.nio.ch.Util$2)
> - locked <0x00000005196432a0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005194ecc58> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d17b000 nid=0x3f81 waiting on condition [0x00007f8443dfc000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051921afd8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f843c01a000 nid=0x3f80 runnable [0x00007f8443efd000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519654460> (a sun.nio.ch.Util$2)
> - locked <0x0000000519654478> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519105410> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08ae800 nid=0x3f7f waiting on condition [0x00007f8443ffe000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519676ae8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f844401a000 nid=0x3f7e runnable [0x00007f85441a6000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051967cb28> (a sun.nio.ch.Util$2)
> - locked <0x000000051967cb40> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005194745e0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d179000 nid=0x3f7d waiting on condition [0x00007f85442a7000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005196432e8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f845801a000 nid=0x3f7c runnable [0x00007f85443a8000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519658440> (a sun.nio.ch.Util$2)
> - locked <0x00000005196583f8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519349128> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f845401a000 nid=0x3f7b runnable [0x00007f85444a9000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519676aa0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519676ab8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051921b020> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f08ad800 nid=0x3f7a waiting on condition [0x00007f85446c5000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519659298> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578463000 nid=0x3f79 waiting on condition [0x00007f85447c6000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005194a3568> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f845c01a000 nid=0x3f78 runnable [0x00007f85445aa000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005196544c0> (a sun.nio.ch.Util$2)
> - locked <0x00000005196544d8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519105458> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d177000 nid=0x3f77 waiting on condition [0x00007f85448c7000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519656020> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f846401a000 nid=0x3f76 runnable [0x00007f85449c8000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519655720> (a sun.nio.ch.Util$2)
> - locked <0x0000000519655738> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519105f98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857115c800 nid=0x3f75 waiting on condition [0x00007f8544ac9000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519402f58> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f847801a000 nid=0x3f74 runnable [0x00007f8544bca000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519654520> (a sun.nio.ch.Util$2)
> - locked <0x0000000519654538> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005191054a0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f847401a000 nid=0x3f73 runnable [0x00007f8544ccb000] 
>    java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.newKeyIterator(HashMap.java:840)
> at java.util.HashMap$KeySet.iterator(HashMap.java:874)
> at java.util.HashSet.iterator(HashSet.java:153)
> at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
> - locked <0x0000000519656248> (a java.util.HashSet)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519656038> (a sun.nio.ch.Util$2)
> - locked <0x0000000519656050> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519104d08> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0015800 nid=0x3f72 waiting on condition [0x00007f8544dcc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051941d010> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578461000 nid=0x3f71 waiting on condition [0x00007f8544ecd000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519477938> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f847c01a000 nid=0x3f70 runnable [0x00007f8544fce000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051942aa08> (a sun.nio.ch.Util$2)
> - locked <0x000000051942aa20> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519349050> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857d175800 nid=0x3f6f waiting on condition [0x00007f85450cf000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519470c60> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f848401a000 nid=0x3f6e runnable [0x00007f85451d0000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519445fa0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519445fb8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005194679d0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570456800 nid=0x3f6d waiting on condition [0x00007f85452d1000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519419230> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f849801a000 nid=0x3f6c runnable [0x00007f85453d2000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519470c78> (a sun.nio.ch.Util$2)
> - locked <0x0000000519470c90> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005194743a0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f849401a000 nid=0x3f6b runnable [0x00007f85454d3000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051945beb8> (a sun.nio.ch.Util$2)
> - locked <0x000000051945bed0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519196000> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0013800 nid=0x3f6a waiting on condition [0x00007f85455d4000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519470cd8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857845f000 nid=0x3f69 waiting on condition [0x00007f85456d5000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051941d1c0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f849c01a000 nid=0x3f68 runnable [0x00007f85457d6000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005194583b8> (a sun.nio.ch.Util$2)
> - locked <0x00000005194583d0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005194743e8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c488000 nid=0x3f67 waiting on condition [0x00007f85458d7000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519424e80> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84a401a000 nid=0x3f66 runnable [0x00007f85459d8000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051941d1d8> (a sun.nio.ch.Util$2)
> - locked <0x000000051941d1f0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519467a18> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570455000 nid=0x3f65 waiting on condition [0x00007f8545ad9000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051945bf18> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84b801a000 nid=0x3f64 runnable [0x00007f8545bda000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519419248> (a sun.nio.ch.Util$2)
> - locked <0x0000000519419200> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519431ee0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84c001a000 nid=0x3f63 runnable [0x00007f8545cdb000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051941d238> (a sun.nio.ch.Util$2)
> - locked <0x000000051941d250> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519467a60> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f0012000 nid=0x3f62 waiting on condition [0x00007f8545ddc000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519458418> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84bc01a000 nid=0x3f61 runnable [0x00007f8545edd000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051945bf30> (a sun.nio.ch.Util$2)
> - locked <0x000000051945bf48> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519196048> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857845e000 nid=0x3f60 waiting on condition [0x00007f8545fde000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519479f10> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c486000 nid=0x3f5f waiting on condition [0x00007f85460df000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051941d298> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84c401a000 nid=0x3f5e runnable [0x00007f85461e0000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051942a9a8> (a sun.nio.ch.Util$2)
> - locked <0x000000051942a9c0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519349098> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570453800 nid=0x3f5d waiting on condition [0x00007f85462e1000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051945bf90> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84cc01a000 nid=0x3f5c runnable [0x00007f85463e2000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519458430> (a sun.nio.ch.Util$2)
> - locked <0x0000000519458448> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519474430> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c484800 nid=0x3f5b waiting on condition [0x00007f85464e3000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051941d2b0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84d401a000 nid=0x3f5a runnable [0x00007f85465e4000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005194191e8> (a sun.nio.ch.Util$2)
> - locked <0x000000051936a4e0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519431f28> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570450800 nid=0x3f59 waiting on condition [0x00007f85466e5000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051943ef60> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84dc01a000 nid=0x3f58 runnable [0x00007f85467e6000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051942a948> (a sun.nio.ch.Util$2)
> - locked <0x000000051942a960> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005193490e0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c482800 nid=0x3f57 waiting on condition [0x00007f85468e7000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051945bfa8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84e401a000 nid=0x3f56 runnable [0x00007f85469e8000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051941d2c8> (a sun.nio.ch.Util$2)
> - locked <0x000000051941d2e0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519467aa8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857044f000 nid=0x3f55 waiting on condition [0x00007f8546ae9000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051945bfc0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84f801a000 nid=0x3f54 runnable [0x00007f8546bea000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051936a4c8> (a sun.nio.ch.Util$2)
> - locked <0x000000051936a480> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519431f70> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84f401a000 nid=0x3f53 runnable [0x00007f8546ceb000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519458490> (a sun.nio.ch.Util$2)
> - locked <0x00000005194584a8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519474478> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578008000 nid=0x3f52 waiting on condition [0x00007f8546dec000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051942a8e8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c00d000 nid=0x3f51 waiting on condition [0x00007f8546eed000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519447098> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f84fc01a000 nid=0x3f50 runnable [0x00007f8546fee000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051941d328> (a sun.nio.ch.Util$2)
> - locked <0x000000051941d340> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519467af0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857044d000 nid=0x3f4f waiting on condition [0x00007f85470ef000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051945bfd8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f851001a000 nid=0x3f4e runnable [0x00007f85471f0000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519424e98> (a sun.nio.ch.Util$2)
> - locked <0x0000000519424e50> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519380d20> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f850c01a000 nid=0x3f4d runnable [0x00007f85472f1000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005194584f0> (a sun.nio.ch.Util$2)
> - locked <0x0000000519458508> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005194744c0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578006800 nid=0x3f4c waiting on condition [0x00007f85473f2000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051945bff0> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c01d800 nid=0x3f4b waiting on condition [0x00007f85474f3000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051936a468> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f851401a000 nid=0x3f4a runnable [0x00007f85475f4000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051941d388> (a sun.nio.ch.Util$2)
> - locked <0x000000051941d3a0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519467b38> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857045b800 nid=0x3f49 waiting on condition [0x00007f85476f5000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051942a900> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f853801a000 nid=0x3f48 runnable [0x00007f85477f6000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519479f28> (a sun.nio.ch.Util$2)
> - locked <0x0000000519479ee0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519203068> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f853001f800 nid=0x3f47 runnable [0x00007f85478f7000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519477950> (a sun.nio.ch.Util$2)
> - locked <0x0000000519477968> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051922faf0> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f852c01f000 nid=0x3f46 runnable [0x00007f85479f8000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005194353f8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519435410> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000000051921b0f8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver I/O Worker 1" daemon prio=10 tid=0x00007f853401a000 nid=0x3f45 runnable [0x00007f8547af9000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051945c008> (a sun.nio.ch.Util$2)
> - locked <0x000000051945c020> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000519196090> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8578005800 nid=0x3f44 waiting on condition [0x00007f8547bfa000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005190de788> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f85f001a800 nid=0x3f43 waiting on condition [0x00007f8547cfb000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518fae040> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f857c022000 nid=0x3f42 waiting on condition [0x00007f8547dfc000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518cafb20> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Avro NettyTransceiver Boss 1" daemon prio=10 tid=0x00007f8570460000 nid=0x3f41 waiting on condition [0x00007f8547efd000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005190db9c8> (a java.util.concurrent.SynchronousQueue$TransferStack)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-2" prio=10 tid=0x00007f8588036000 nid=0x3f40 runnable [0x00007f8547ffe000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005190ee0e8> (a sun.nio.ch.Util$2)
> - locked <0x00000005191f0be8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005190e1d18> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-140-thread-1" prio=10 tid=0x00007f858801b800 nid=0x3f3f runnable [0x00007f858c18a000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000518f16658> (a sun.nio.ch.Util$2)
> - locked <0x000000051941f820> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005190edc80> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-139-thread-1" prio=10 tid=0x00007f8580002800 nid=0x3f3e runnable [0x00007f858c28b000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x000000051913eb50> (a sun.nio.ch.Util$2)
> - locked <0x000000051913eca0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000518c9c4e8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-148-thread-1" prio=10 tid=0x00007f856844c000 nid=0x3f3d waiting on condition [0x00007f858c38c000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x00000005190cfe98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-146-thread-1" prio=10 tid=0x00007f85dc006800 nid=0x3f3c waiting on condition [0x00007f858c48d000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519018558> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-147-thread-1" prio=10 tid=0x00007f85e82b1000 nid=0x3f3b waiting on condition [0x00007f858c58e000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051903b488> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-145-thread-1" prio=10 tid=0x00007f8574027000 nid=0x3f3a waiting on condition [0x00007f858c68f000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051903b998> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-141-thread-1" prio=10 tid=0x00007f85e8159000 nid=0x3f39 runnable [0x00007f861cd8e000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005190db9e0> (a sun.nio.ch.Util$2)
> - locked <0x00000005190db998> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000518c9c020> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-138-thread-1" prio=10 tid=0x00007f856844a800 nid=0x3f38 runnable [0x00007f861cf90000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00000005190e1b68> (a sun.nio.ch.Util$2)
> - locked <0x00000005190e1b80> (a java.util.Collections$UnmodifiableSet)
> - locked <0x0000000518c9c608> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-143-thread-1" prio=10 tid=0x00007f85dc003000 nid=0x3f37 runnable [0x00007f861d091000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519018fa8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519018fc0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005190e2228> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "pool-137-thread-1" prio=10 tid=0x00007f8574007000 nid=0x3f36 runnable [0x00007f861ce8f000] 
>    java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x0000000519003dc8> (a sun.nio.ch.Util$2)
> - locked <0x0000000519003de0> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000005190ea1a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:219)
> at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 
> "1407844613@qtp-1263282395-1 - Acceptor0 SocketConnector@0.0.0.0:54321 (http://SocketConnector@0.0.0.0:54321)" prio=10 tid=0x00007f85e404a800 nid=0x3f35 runnable [0x00007f858c790000] 
>    java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
> - locked <0x00000005190e2720> (a java.net.SocksSocketImpl)
> at java.net.ServerSocket.implAccept(ServerSocket.java:462)
> at java.net.ServerSocket.accept(ServerSocket.java:430)
> at org.mortbay.jetty.bio.SocketConnector.accept(SocketConnector.java:99)
> at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> 
> "208924888@qtp-1263282395-0" prio=10 tid=0x00007f85e4021800 nid=0x3f34 in Object.wait() [0x00007f861c15b000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000005190d0528> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> - locked <0x00000005190d0528> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
> 
> "lifecycleSupervisor-1-9" prio=10 tid=0x00007f85e401a800 nid=0x3f33 waiting on condition [0x00007f861c687000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "lifecycleSupervisor-1-8" prio=10 tid=0x00007f85e4018800 nid=0x3f32 waiting on condition [0x00007f861c788000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "lifecycleSupervisor-1-7" prio=10 tid=0x00007f85e4016800 nid=0x3f31 waiting on condition [0x00007f861c889000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "lifecycleSupervisor-1-6" prio=10 tid=0x00007f85e4014800 nid=0x3f30 waiting on condition [0x00007f861c98a000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "lifecycleSupervisor-1-5" prio=10 tid=0x00007f85e4013000 nid=0x3f2f waiting on condition [0x00007f861ca8b000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "lifecycleSupervisor-1-4" prio=10 tid=0x00007f85e4028000 nid=0x3f2e waiting on condition [0x00007f861cb8c000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "lifecycleSupervisor-1-3" prio=10 tid=0x00007f85e4026800 nid=0x3f2d waiting on condition [0x00007f861cc8d000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Log-BackgroundWorker-ch1" prio=10 tid=0x00007f85dc001000 nid=0x3f27 waiting on condition [0x00007f861d299000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000519017718> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Log-BackgroundWorker-ch2" prio=10 tid=0x00007f85e80ca000 nid=0x3f26 waiting on condition [0x00007f861d39a000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000000051903b4e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "lifecycleSupervisor-1-2" prio=10 tid=0x00007f85e4025000 nid=0x3f25 waiting on condition [0x00007f861d6ba000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "lifecycleSupervisor-1-1" prio=10 tid=0x00007f85e400e800 nid=0x3f24 waiting on condition [0x00007f861d7bb000] 
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "conf-file-poller-0" prio=10 tid=0x00007f85f0001000 nid=0x3f23 waiting on condition [0x00007f861d8bc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518ec72f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "DestroyJavaVM" prio=10 tid=0x00007f86a800c800 nid=0x3efa waiting on condition [0x0000000000000000] 
>    java.lang.Thread.State: RUNNABLE
> 
> "lifecycleSupervisor-1-0" prio=10 tid=0x00007f86a89f5800 nid=0x3f22 waiting on condition [0x00007f861d9bd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0000000518c85670> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
> 
> "Low Memory Detector" daemon prio=10 tid=0x00007f86a831d800 nid=0x3f20 runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "C2 CompilerThread1" daemon prio=10 tid=0x00007f86a831b000 nid=0x3f1f waiting on condition [0x0000000000000000] 
>    java.lang.Thread.State: RUNNABLE
> 
> "C2 CompilerThread0" daemon prio=10 tid=0x00007f86a8319000 nid=0x3f1e waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "Signal Dispatcher" daemon prio=10 tid=0x00007f86a8316800 nid=0x3f1d runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> 
> "Surrogate Locker Thread (Concurrent GC)" daemon prio=10 tid=0x00007f86a8314800 nid=0x3f1c waiting on condition [0x0000000000000000] 
>    java.lang.Thread.State: RUNNABLE
> 
> "Finalizer" daemon prio=10 tid=0x00007f86a82f9800 nid=0x3f1b in Object.wait() [0x00007f861e3eb000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0000000518c843c0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> - locked <0x0000000518c843c0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> 
> "Reference Handler" daemon prio=10 tid=0x00007f86a82f7800 nid=0x3f1a in Object.wait() [0x00007f861e4ec000]
>    java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0000000518c80860> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> - locked <0x0000000518c80860> (a java.lang.ref.Reference$Lock)
> 
> "VM Thread" prio=10 tid=0x00007f86a82f1000 nid=0x3f19 runnable 
> 
> "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f86a801b800 nid=0x3efb runnable  
> 
> "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f86a801d000 nid=0x3efc runnable 
> 
> "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f86a801f000 nid=0x3efd runnable  
> 
> "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f86a8021000 nid=0x3efe runnable 
> 
> "Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x00007f86a8022800 nid=0x3eff runnable  
> 
> "Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x00007f86a8024800 nid=0x3f00 runnable 
> 
> "Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x00007f86a8026000 nid=0x3f01 runnable  
> 
> "Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x00007f86a8028000 nid=0x3f02 runnable 
> 
> "Gang worker#8 (Parallel GC Threads)" prio=10 tid=0x00007f86a802a000 nid=0x3f03 runnable  
> 
> "Gang worker#9 (Parallel GC Threads)" prio=10 tid=0x00007f86a802b800 nid=0x3f04 runnable 
> 
> "Gang worker#10 (Parallel GC Threads)" prio=10 tid=0x00007f86a802d800 nid=0x3f05 runnable  
> 
> "Gang worker#11 (Parallel GC Threads)" prio=10 tid=0x00007f86a802f800 nid=0x3f06 runnable 
> 
> "Gang worker#12 (Parallel GC Threads)" prio=10 tid=0x00007f86a8031000 nid=0x3f07 runnable  
> 
> "Gang worker#13 (Parallel GC Threads)" prio=10 tid=0x00007f86a8033000 nid=0x3f08 runnable 
> 
> "Gang worker#14 (Parallel GC Threads)" prio=10 tid=0x00007f86a8035000 nid=0x3f09 runnable  
> 
> "Gang worker#15 (Parallel GC Threads)" prio=10 tid=0x00007f86a8036800 nid=0x3f0a runnable 
> 
> "Gang worker#16 (Parallel GC Threads)" prio=10 tid=0x00007f86a8038800 nid=0x3f0b runnable  
> 
> "Gang worker#17 (Parallel GC Threads)" prio=10 tid=0x00007f86a803a000 nid=0x3f0c runnable 
> 
> "Gang worker#18 (Parallel GC Threads)" prio=10 tid=0x00007f86a803c000 nid=0x3f0d runnable  
> 
> "Gang worker#19 (Parallel GC Threads)" prio=10 tid=0x00007f86a803e000 nid=0x3f0e runnable 
> 
> "Gang worker#20 (Parallel GC Threads)" prio=10 tid=0x00007f86a803f800 nid=0x3f0f runnable  
> 
> "Gang worker#21 (Parallel GC Threads)" prio=10 tid=0x00007f86a8041800 nid=0x3f10 runnable 
> 
> "Gang worker#22 (Parallel GC Threads)" prio=10 tid=0x00007f86a8043800 nid=0x3f11 runnable  
> 
> "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f86a82aa000 nid=0x3f18 runnable 
> "Gang worker#0 (Parallel CMS Threads)" prio=10 tid=0x00007f86a829e800 nid=0x3f12 runnable 
> 
> "Gang worker#1 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a0000 nid=0x3f13 runnable 
> 
> "Gang worker#2 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a2000 nid=0x3f14 runnable  
> 
> "Gang worker#3 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a3800 nid=0x3f15 runnable 
> 
> "Gang worker#4 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a5800 nid=0x3f16 runnable  
> 
> "Gang worker#5 (Parallel CMS Threads)" prio=10 tid=0x00007f86a82a7800 nid=0x3f17 runnable 
> 
> "VM Periodic Task Thread" prio=10 tid=0x00007f86a8329000 nid=0x3f21 waiting on condition  
> 
> JNI global references: 1749
> 
>  
> 
> -- 
> 
> P | (415) 677-9222 ext. 205 F | (415) 677-0895 | pankaj@brightroll.com (mailto:pankaj@brightroll.com) 
> Pankaj Gupta | Software Engineer
> BrightRoll, Inc. | Smart Video Advertising | www.brightroll.com (http://www.brightroll.com/)
> 
> United States | Canada | United Kingdom | Germany 
> 
> We're hiring (http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7)!