flume agent get OOME from downstream flume tier

[Shangan Chen <ch...@gmail.com>]

Sometime, I got OOME from flume-agent down-streaming events to
flume-collector, I try to debug what cause the exception. I can only get
the java stack in flume-agent，and when I try to find the corresponding log
in flume-collector, it seems OK, just some connection state log like
DISCONNECTED,UNBOUND, etc.  Has anybody met the same problem and can
explain ?

*log in flume-agent: *

28 Apr 2014 17:48:33,561 WARN
 [SinkRunner-PollingRunner-LoadBalancingSinkProcessor]
(org.apache.flume.sink.LoadBalancingSinkProcessor.process:173)  - Sink
failed to consume event. Attempting next sink if available.
org.apache.flume.EventDeliveryException: Failed to send events
        at
org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:383)
        at
org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:169)
        at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
host: data-log10, port: 4400 }: Failed to send batch
        at
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:294)
        at
org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:367)
        ... 3 more
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
host: data-log10, port: 4400 }: Exception thrown from remote handler
        at
org.apache.flume.api.NettyAvroRpcClient.waitForStatusOK(NettyAvroRpcClient.java:376)
        at
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:353)
        at
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:282)
        ... 4 more
Caused by: java.util.concurrent.ExecutionException:
java.lang.OutOfMemoryError: Direct buffer memory
        at org.apache.avro.ipc.CallFuture.get(CallFuture.java:128)
        at
org.apache.flume.api.NettyAvroRpcClient.waitForStatusOK(NettyAvroRpcClient.java:368)
        ... 6 more
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:632)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:97)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
        at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:155)
        at sun.nio.ch.IOUtil.write(IOUtil.java:33)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:336)
        at
org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:198)
        at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:468)
        at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:423)
        at
org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:364)
        at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.processWriteTaskQueue(AbstractNioWorker.java:341)
        at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:237)
        at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        ... 1 more



*log in flume-collector*

28 Apr 2014 17:48:33,570 INFO
 [pool-27-thread-13]
(org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream:171)
 - [id: 0x13ba880d, /10.64.22.177:50867 :> /10.64.20.209:4400] DISCONNECTED
28 Apr 2014 17:48:33,571 INFO
 [pool-27-thread-13]
(org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream:171)
 - [id: 0x13ba880d, /10.64.22.177:50867 :> /10.64.20.209:4400] UNBOUND
28 Apr 2014 17:48:33,571 INFO
 [pool-27-thread-13]
(org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream:171)
 - [id: 0x13ba880d, /10.64.22.177:50867 :> /10.64.20.209:4400] CLOSED
28 Apr 2014 17:48:33,571 INFO
 [pool-27-thread-13]
(org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.channelClosed:209)
 - Connection to /10.64.22.177:50867 disconnected.


-- 
have a good day!
chenshang'an