You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by "Yanzhi.liu" <40...@qq.com> on 2012/10/16 05:08:31 UTC
回复： About file channel yum error
Hello Brock:
My flume's configuration is
agent_foo.sources = tailsource-1
agent_foo.channels = fileChannel
agent_foo.sinks = hdfsSink-1
agent_foo.sources.tailsource-1.type = com.chinacache.cpis.sources.CpisAvroSource
agent_foo.sources.tailsource-1.channels=fileChannel
agent_foo.sources.tailsource-1.bind=221.130.18.90
agent_foo.sources.tailsource-1.port=4545
#agent_foo.sources.tailsource-1.threads=10
agent_foo.channels.fileChannel.type = file
agent_foo.channels.fileChannel.checkpointDir=/home/hadoop/checkpoint
agent_foo.channels.fileChannel.dataDirs=/data/cache2/data
agent_foo.channels.fileChannel.maxFileSize=524288000
#agent_foo.channels.fileChannel.capacity = 10000
agent_foo.sinks.hdfsSink-1.type = hdfs
agent_foo.sinks.hdfsSink-1.channel = fileChannel
agent_foo.sinks.hdfsSink-1.hdfs.path = hdfs://CMN-NJ-2-579:9000/user/hadoop/fc_logs/%{month}/%{day}/%{deviceId}/%{hour}
#agent_foo.sinks.hdfsSink-1.hdfs.path = hdfs://CMN-NJ-2-579:9000/user/hadoop/fc_logs/%{month}/%{day}/%{hour}/%{deviceId}
agent_foo.sinks.hdfsSink-1.hdfs.filePrefix = cpisfc-assess.log
agent_foo.sinks.hdfsSink-1.hdfs.rollInterval=600
agent_foo.sinks.hdfsSink-1.hdfs.rollCount=0
agent_foo.sinks.hdfsSink-1.hdfs.rollSize = 506870912
agent_foo.sinks.hdfsSink-1.hdfs.fileType=CompressedStream
agent_foo.sinks.hdfsSink-1.hdfs.codeC=gzip
agent_foo.sinks.hdfsSink-1.hdfs.writeFormat=Text
agent_foo.sinks.hdfsSink-1.hdfs.batchSize=1
agent_foo.sinks.hdfsSink-1.serializer=avro_event
And the java configuration was JAVA_OPTS="-Xms100m -Xmx2048m -Dcom.sun.management.jmxremote";
Thank you very much!
My Name :
Yanzhi Liu
------------------ 原始邮件 ------------------
发件人: "Brock Noland"<br...@cloudera.com>;
发送时间: 2012年10月16日(星期二) 中午11:01
收件人: "user"<us...@flume.apache.org>;
主题: Re: About file channel yum error
What is your channel capacity and configured agent heap size?
On Monday, October 15, 2012 at 9:19 PM, Yanzhi.liu wrote:
Hello everyone:
I am using the flume 1.3.0.But the flume clusters was working after two days and the flume was stoped. And the flume.log was only error about the:
14 Oct 2012 15:29:17,018 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.HDFSEventSink.process:448) - process failed
java.lang.OutOfMemoryError
at java.io.RandomAccessFile.readBytes(Native Method)
at java.io.RandomAccessFile.read(RandomAccessFile.java:338)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:397)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377)
at org.apache.flume.channel.file.LogFile.readDelimitedBuffer(LogFile.java:486)
at org.apache.flume.channel.file.LogFileV3$RandomReader.doGet(LogFileV3.java:258)
at org.apache.flume.channel.file.LogFile$RandomReader.get(LogFile.java:298)
at org.apache.flume.channel.file.Log.get(Log.java:409)
at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:447)
at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:91)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:387)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)
So I want to how to avoid the error.
Thanks very much!
My Name:
Yanzhi Liu
Re: 回复： About file channel yum error
Posted by Kevin Lee <de...@yottaa.com>.
Hi,
Today Amazon has EBS outage, maybe it's a not flume issue, thanks you.
-Kevin
On Oct 23, 2012, at 10:31 AM, Brock Noland wrote:
> Hi,
>
> Sorry that thread dump doesn't provide much. If you use kill -3 <pid> it prints heap information at the bottom of the of the dump. Also the OOM error message long with a configuration would be helpful.
>
> Brock
>
> On Mon, Oct 22, 2012 at 9:26 PM, Kevin Lee <de...@yottaa.com> wrote:
> Hey,
>
> I got a OOM error too, and the detail as follow:
>
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> flume 22902 59.6 23.5 2813560 1802672 ? Sl Oct18 4210:34 /usr/java/jdk1.6.0_33/bin/java -Xms500m -Xmx1500m -cp /etc/flume-ng/conf:/usr/lib/flume-ng/lib/*:/etc/hadoop/conf:/usr/lib/hadoop/lib/activation-1.1.jar
>
> free -m
> total used free shared buffers cached
> Mem: 7467 7430 36 0 13 5011
> -/+ buffers/cache: 2405 5061
> Swap: 4095 15 4080
>
> thread dump:
>
> /usr/java/default/bin/jstack -F 22902
> Attaching to process ID 22902, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 20.8-b03
> Deadlock Detection:
>
> No deadlocks found.
>
> Thread 11929: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=196 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2025 (Compiled frame)
> - java.util.concurrent.DelayQueue.take() @bci=57, line=164 (Compiled frame)
> - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609 (Compiled frame)
> - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23734: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23733: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23732: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23731: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23730: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23729: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23728: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23727: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23726: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23725: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.DelayQueue.take() @bci=28, line=160 (Compiled frame)
> - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609 (Compiled frame)
> - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23723: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23722: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23721: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23720: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23719: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23718: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23717: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23716: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23715: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23714: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.DelayQueue.take() @bci=28, line=160 (Compiled frame)
> - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609 (Compiled frame)
> - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602 (Interpreted frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Interpreted frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23705: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23702: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Interpreted frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Interpreted frame)
> - java.util.concurrent.DelayQueue.take() @bci=28, line=160 (Interpreted frame)
> - org.apache.hadoop.fs.DelegationTokenRenewer.run() @bci=6, line=152 (Interpreted frame)
>
>
> Thread 23700: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23696: (state = IN_NATIVE)
> - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
> - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=210 (Compiled frame)
> - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=65 (Compiled frame)
> - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=69 (Compiled frame)
> - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80 (Compiled frame)
> - org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run() @bci=23, line=219 (Compiled frame)
> - org.jboss.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23693: (state = IN_NATIVE)
> - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
> - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=210 (Compiled frame)
> - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=65 (Compiled frame)
> - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=69 (Compiled frame)
> - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80 (Compiled frame)
> - org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run() @bci=23, line=219 (Compiled frame)
> - org.jboss.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23666: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=196 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2025 (Compiled frame)
> - java.util.concurrent.DelayQueue.take() @bci=57, line=164 (Compiled frame)
> - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609 (Compiled frame)
> - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23662: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=196 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2025 (Compiled frame)
> - java.util.concurrent.DelayQueue.take() @bci=57, line=164 (Compiled frame)
> - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609 (Compiled frame)
> - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23660: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.DelayQueue.take() @bci=28, line=160 (Compiled frame)
> - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609 (Compiled frame)
> - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23651: (state = BLOCKED)
>
>
> Thread 23650: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
> - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Interpreted frame)
> - java.lang.ref.ReferenceQueue.remove() @bci=2, line=134 (Interpreted frame)
> - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Interpreted frame)
>
>
> Thread 23649: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
> - java.lang.Object.wait() @bci=2, line=485 (Compiled frame)
> - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 (Compiled frame)
>
>
> Thread 23645: (state = BLOCKED)
> - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
> - org.apache.flume.lifecycle.LifecycleController.waitForOneOf(org.apache.flume.lifecycle.LifecycleAware, org.apache.flume.lifecycle.LifecycleState[], long) @bci=109, line=71 (Compiled frame)
> - org.apache.flume.lifecycle.LifecycleController.waitForOneOf(org.apache.flume.lifecycle.LifecycleAware, org.apache.flume.lifecycle.LifecycleState[]) @bci=3, line=49 (Interpreted frame)
> - org.apache.flume.node.Application.run() @bci=154, line=167 (Interpreted frame)
> - org.apache.flume.node.Application.main(java.lang.String[]) @bci=21, line=70 (Interpreted frame)
>
>
>
> On Oct 16, 2012, at 11:10 PM, Brock Noland wrote:
>
>> When the agent sees an OOM, can you take a thread dump (kill -3 <pid>) and post it on pastebin? Please be sure to include the heap summary section at the bottom of the dump.
>>
>> Brock
>>
>> On Mon, Oct 15, 2012 at 10:08 PM, Yanzhi.liu <40...@qq.com> wrote:
>> Hello Brock:
>> My flume's configuration is
>> agent_foo.sources = tailsource-1
>> agent_foo.channels = fileChannel
>> agent_foo.sinks = hdfsSink-1
>> agent_foo.sources.tailsource-1.type = com.chinacache.cpis.sources.CpisAvroSource
>> agent_foo.sources.tailsource-1.channels=fileChannel
>> agent_foo.sources.tailsource-1.bind=221.130.18.90
>> agent_foo.sources.tailsource-1.port=4545
>> #agent_foo.sources.tailsource-1.threads=10
>> agent_foo.channels.fileChannel.type = file
>> agent_foo.channels.fileChannel.checkpointDir=/home/hadoop/checkpoint
>> agent_foo.channels.fileChannel.dataDirs=/data/cache2/data
>> agent_foo.channels.fileChannel.maxFileSize=524288000
>> #agent_foo.channels.fileChannel.capacity = 10000
>> agent_foo.sinks.hdfsSink-1.type = hdfs
>> agent_foo.sinks.hdfsSink-1.channel = fileChannel
>> agent_foo.sinks.hdfsSink-1.hdfs.path = hdfs://CMN-NJ-2-579:9000/user/hadoop/fc_logs/%{month}/%{day}/%{deviceId}/%{hour}
>> #agent_foo.sinks.hdfsSink-1.hdfs.path = hdfs://CMN-NJ-2-579:9000/user/hadoop/fc_logs/%{month}/%{day}/%{hour}/%{deviceId}
>> agent_foo.sinks.hdfsSink-1.hdfs.filePrefix = cpisfc-assess.log
>> agent_foo.sinks.hdfsSink-1.hdfs.rollInterval=600
>> agent_foo.sinks.hdfsSink-1.hdfs.rollCount=0
>> agent_foo.sinks.hdfsSink-1.hdfs.rollSize = 506870912
>> agent_foo.sinks.hdfsSink-1.hdfs.fileType=CompressedStream
>> agent_foo.sinks.hdfsSink-1.hdfs.codeC=gzip
>> agent_foo.sinks.hdfsSink-1.hdfs.writeFormat=Text
>> agent_foo.sinks.hdfsSink-1.hdfs.batchSize=1
>> agent_foo.sinks.hdfsSink-1.serializer=avro_event
>> And the java configuration was JAVA_OPTS="-Xms100m -Xmx2048m -Dcom.sun.management.jmxremote";
>> Thank you very much!
>> My Name :
>> Yanzhi Liu
>>
>>
>> ------------------ 原始邮件 ------------------
>> 发件人: "Brock Noland"<br...@cloudera.com>;
>> 发送时间: 2012年10月16日(星期二) 中午11:01
>> 收件人: "user"<us...@flume.apache.org>;
>> 主题: Re: About file channel yum error
>>
>> What is your channel capacity and configured agent heap size?
>>
>> On Monday, October 15, 2012 at 9:19 PM, Yanzhi.liu wrote:
>>
>>> Hello everyone:
>>> I am using the flume 1.3.0.But the flume clusters was working after two days and the flume was stoped. And the flume.log was only error about the:
>>> 14 Oct 2012 15:29:17,018 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.HDFSEventSink.process:448) - process failed
>>> java.lang.OutOfMemoryError
>>> at java.io.RandomAccessFile.readBytes(Native Method)
>>> at java.io.RandomAccessFile.read(RandomAccessFile.java:338)
>>> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:397)
>>> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377)
>>> at org.apache.flume.channel.file.LogFile.readDelimitedBuffer(LogFile.java:486)
>>> at org.apache.flume.channel.file.LogFileV3$RandomReader.doGet(LogFileV3.java:258)
>>> at org.apache.flume.channel.file.LogFile$RandomReader.get(LogFile.java:298)
>>> at org.apache.flume.channel.file.Log.get(Log.java:409)
>>> at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:447)
>>> at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>> at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:91)
>>> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:387)
>>> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>> at java.lang.Thread.run(Thread.java:662)
>>> So I want to how to avoid the error.
>>> Thanks very much!
>>> My Name:
>>> Yanzhi Liu
>>
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: 回复： About file channel yum error
Posted by Brock Noland <br...@cloudera.com>.
Hi,
Sorry that thread dump doesn't provide much. If you use kill -3 <pid> it
prints heap information at the bottom of the of the dump. Also the OOM
error message long with a configuration would be helpful.
Brock
On Mon, Oct 22, 2012 at 9:26 PM, Kevin Lee <de...@yottaa.com> wrote:
> Hey,
>
> I got a OOM error too, and the detail as follow:
>
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> flume 22902 59.6 23.5 2813560 1802672 ? Sl Oct18 4210:34
> /usr/java/jdk1.6.0_33/bin/java -Xms500m -Xmx1500m -cp
> /etc/flume-ng/conf:/usr/lib/flume-ng/lib/*:/etc/hadoop/conf:/usr/lib/hadoop/lib/activation-1.1.jar
>
> free -m
> total used free shared buffers cached
> Mem: 7467 7430 36 0 13 5011
> -/+ buffers/cache: 2405 5061
> Swap: 4095 15 4080
>
> thread dump:
>
> /usr/java/default/bin/jstack -F 22902
> Attaching to process ID 22902, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 20.8-b03
> Deadlock Detection:
>
> No deadlocks found.
>
> Thread 11929: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object,
> long) @bci=20, line=196 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
> @bci=68, line=2025 (Compiled frame)
> - java.util.concurrent.DelayQueue.take() @bci=57, line=164 (Compiled
> frame)
> -
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
> @bci=4, line=609 (Compiled frame)
> -
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
> @bci=1, line=602 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23734: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23733: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23732: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23731: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23730: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23729: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23728: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23727: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23726: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23725: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.DelayQueue.take() @bci=28, line=160 (Compiled
> frame)
> -
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
> @bci=4, line=609 (Compiled frame)
> -
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
> @bci=1, line=602 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23723: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23722: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23721: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23720: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23719: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23718: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23717: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23716: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23715: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23714: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.DelayQueue.take() @bci=28, line=160 (Compiled
> frame)
> -
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
> @bci=4, line=609 (Compiled frame)
> -
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
> @bci=1, line=602 (Interpreted frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Interpreted frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23705: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23702: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Interpreted frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Interpreted frame)
> - java.util.concurrent.DelayQueue.take() @bci=28, line=160 (Interpreted
> frame)
> - org.apache.hadoop.fs.DelegationTokenRenewer.run() @bci=6, line=152
> (Interpreted frame)
>
>
> Thread 23700: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23696: (state = IN_NATIVE)
> - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0
> (Compiled frame; information may be imprecise)
> - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=210 (Compiled
> frame)
> - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=65 (Compiled
> frame)
> - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=69
> (Compiled frame)
> - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80 (Compiled frame)
> -
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run()
> @bci=23, line=219 (Compiled frame)
> - org.jboss.netty.util.internal.DeadLockProofWorker$1.run() @bci=14,
> line=42 (Interpreted frame)
> -
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable)
> @bci=59, line=886 (Interpreted frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908
> (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23693: (state = IN_NATIVE)
> - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0
> (Compiled frame; information may be imprecise)
> - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=210 (Compiled
> frame)
> - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=65 (Compiled
> frame)
> - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=69
> (Compiled frame)
> - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80 (Compiled frame)
> -
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run()
> @bci=23, line=219 (Compiled frame)
> - org.jboss.netty.util.internal.DeadLockProofWorker$1.run() @bci=14,
> line=42 (Interpreted frame)
> -
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable)
> @bci=59, line=886 (Interpreted frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908
> (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23666: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object,
> long) @bci=20, line=196 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
> @bci=68, line=2025 (Compiled frame)
> - java.util.concurrent.DelayQueue.take() @bci=57, line=164 (Compiled
> frame)
> -
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
> @bci=4, line=609 (Compiled frame)
> -
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
> @bci=1, line=602 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23662: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object,
> long) @bci=20, line=196 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
> @bci=68, line=2025 (Compiled frame)
> - java.util.concurrent.DelayQueue.take() @bci=57, line=164 (Compiled
> frame)
> -
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
> @bci=4, line=609 (Compiled frame)
> -
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
> @bci=1, line=602 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23660: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.DelayQueue.take() @bci=28, line=160 (Compiled
> frame)
> -
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
> @bci=4, line=609 (Compiled frame)
> -
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
> @bci=1, line=602 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23651: (state = BLOCKED)
>
>
> Thread 23650: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
> imprecise)
> - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118
> (Interpreted frame)
> - java.lang.ref.ReferenceQueue.remove() @bci=2, line=134 (Interpreted
> frame)
> - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159
> (Interpreted frame)
>
>
> Thread 23649: (state = BLOCKED)
> - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
> imprecise)
> - java.lang.Object.wait() @bci=2, line=485 (Compiled frame)
> - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116
> (Compiled frame)
>
>
> Thread 23645: (state = BLOCKED)
> - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be
> imprecise)
> -
> org.apache.flume.lifecycle.LifecycleController.waitForOneOf(org.apache.flume.lifecycle.LifecycleAware,
> org.apache.flume.lifecycle.LifecycleState[], long) @bci=109, line=71
> (Compiled frame)
> -
> org.apache.flume.lifecycle.LifecycleController.waitForOneOf(org.apache.flume.lifecycle.LifecycleAware,
> org.apache.flume.lifecycle.LifecycleState[]) @bci=3, line=49 (Interpreted
> frame)
> - org.apache.flume.node.Application.run() @bci=154, line=167 (Interpreted
> frame)
> - org.apache.flume.node.Application.main(java.lang.String[]) @bci=21,
> line=70 (Interpreted frame)
>
>
>
> On Oct 16, 2012, at 11:10 PM, Brock Noland wrote:
>
> When the agent sees an OOM, can you take a thread dump (kill -3 <pid>) and
> post it on pastebin? Please be sure to include the heap summary section at
> the bottom of the dump.
>
> Brock
>
> On Mon, Oct 15, 2012 at 10:08 PM, Yanzhi.liu <40...@qq.com> wrote:
>
>> Hello Brock:
>> My flume's configuration is
>> agent_foo.sources = tailsource-1
>> agent_foo.channels = fileChannel
>> agent_foo.sinks = hdfsSink-1
>> agent_foo.sources.tailsource-1.type =
>> com.chinacache.cpis.sources.CpisAvroSource
>> agent_foo.sources.tailsource-1.channels=fileChannel
>> agent_foo.sources.tailsource-1.bind=221.130.18.90
>> agent_foo.sources.tailsource-1.port=4545
>> #agent_foo.sources.tailsource-1.threads=10
>> agent_foo.channels.fileChannel.type = file
>> agent_foo.channels.fileChannel.checkpointDir=/home/hadoop/checkpoint
>> agent_foo.channels.fileChannel.dataDirs=/data/cache2/data
>> agent_foo.channels.fileChannel.maxFileSize=524288000
>> #agent_foo.channels.fileChannel.capacity = 10000
>> agent_foo.sinks.hdfsSink-1.type = hdfs
>> agent_foo.sinks.hdfsSink-1.channel = fileChannel
>> agent_foo.sinks.hdfsSink-1.hdfs.path =
>> hdfs://CMN-NJ-2-579:9000/user/hadoop/fc_logs/%{month}/%{day}/%{deviceId}/%{hour}
>> #agent_foo.sinks.hdfsSink-1.hdfs.path =
>> hdfs://CMN-NJ-2-579:9000/user/hadoop/fc_logs/%{month}/%{day}/%{hour}/%{deviceId}
>> agent_foo.sinks.hdfsSink-1.hdfs.filePrefix = cpisfc-assess.log
>> agent_foo.sinks.hdfsSink-1.hdfs.rollInterval=600
>> agent_foo.sinks.hdfsSink-1.hdfs.rollCount=0
>> agent_foo.sinks.hdfsSink-1.hdfs.rollSize = 506870912
>> agent_foo.sinks.hdfsSink-1.hdfs.fileType=CompressedStream
>> agent_foo.sinks.hdfsSink-1.hdfs.codeC=gzip
>> agent_foo.sinks.hdfsSink-1.hdfs.writeFormat=Text
>> agent_foo.sinks.hdfsSink-1.hdfs.batchSize=1
>> agent_foo.sinks.hdfsSink-1.serializer=avro_event
>> And the java configuration was JAVA_OPTS="-Xms100m -Xmx2048m
>> -Dcom.sun.management.jmxremote";
>> Thank you very much!
>> My Name :
>> Yanzhi Liu
>> **
>>
>>
>> ------------------ 原始邮件 ------------------
>> *发件人:* "Brock Noland"<br...@cloudera.com>;
>> *发送时间:* 2012年10月16日(星期二) 中午11:01
>> *收件人:* "user"<us...@flume.apache.org>; **
>> *主题:* Re: About file channel yum error
>>
>> What is your channel capacity and configured agent heap size?
>>
>> On Monday, October 15, 2012 at 9:19 PM, Yanzhi.liu wrote:
>>
>> Hello everyone:
>> I am using the flume 1.3.0.But the flume clusters was working after
>> two days and the flume was stoped. And the flume.log was only error about
>> the:
>> 14 Oct 2012 15:29:17,018 ERROR
>> [SinkRunner-PollingRunner-DefaultSinkProcessor]
>> (org.apache.flume.sink.hdfs.HDFSEventSink.process:448) - process failed
>> java.lang.OutOfMemoryError
>> at java.io.RandomAccessFile.readBytes(Native Method)
>> at java.io.RandomAccessFile.read(RandomAccessFile.java:338)
>> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:397)
>> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377)
>> at
>> org.apache.flume.channel.file.LogFile.readDelimitedBuffer(LogFile.java:486)
>> at
>> org.apache.flume.channel.file.LogFileV3$RandomReader.doGet(LogFileV3.java:258)
>> at
>> org.apache.flume.channel.file.LogFile$RandomReader.get(LogFile.java:298)
>> at org.apache.flume.channel.file.Log.get(Log.java:409)
>> at
>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:447)
>> at
>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>> at
>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:91)
>> at
>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:387)
>> at
>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>> at
>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>> at java.lang.Thread.run(Thread.java:662)
>> So I want to how to avoid the error.
>> Thanks very much!
>> My Name:
>> Yanzhi Liu
>>
>>
>> **
>>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: 回复： About file channel yum error
Posted by Kevin Lee <de...@yottaa.com>.
Hey,
I got a OOM error too, and the detail as follow:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
flume 22902 59.6 23.5 2813560 1802672 ? Sl Oct18 4210:34 /usr/java/jdk1.6.0_33/bin/java -Xms500m -Xmx1500m -cp /etc/flume-ng/conf:/usr/lib/flume-ng/lib/*:/etc/hadoop/conf:/usr/lib/hadoop/lib/activation-1.1.jar
free -m
total used free shared buffers cached
Mem: 7467 7430 36 0 13 5011
-/+ buffers/cache: 2405 5061
Swap: 4095 15 4080
thread dump:
/usr/java/default/bin/jstack -F 22902
Attaching to process ID 22902, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 20.8-b03
Deadlock Detection:
No deadlocks found.
Thread 11929: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=196 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2025 (Compiled frame)
- java.util.concurrent.DelayQueue.take() @bci=57, line=164 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23734: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23733: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23732: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23731: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23730: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23729: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23728: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23727: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23726: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23725: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.DelayQueue.take() @bci=28, line=160 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23723: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23722: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23721: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23720: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23719: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23718: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23717: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23716: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23715: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23714: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.DelayQueue.take() @bci=28, line=160 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23705: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23702: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Interpreted frame)
- java.util.concurrent.DelayQueue.take() @bci=28, line=160 (Interpreted frame)
- org.apache.hadoop.fs.DelegationTokenRenewer.run() @bci=6, line=152 (Interpreted frame)
Thread 23700: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23696: (state = IN_NATIVE)
- sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
- sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=210 (Compiled frame)
- sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=65 (Compiled frame)
- sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=69 (Compiled frame)
- sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80 (Compiled frame)
- org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run() @bci=23, line=219 (Compiled frame)
- org.jboss.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23693: (state = IN_NATIVE)
- sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
- sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=210 (Compiled frame)
- sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=65 (Compiled frame)
- sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=69 (Compiled frame)
- sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80 (Compiled frame)
- org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run() @bci=23, line=219 (Compiled frame)
- org.jboss.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23666: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=196 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2025 (Compiled frame)
- java.util.concurrent.DelayQueue.take() @bci=57, line=164 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23662: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=196 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2025 (Compiled frame)
- java.util.concurrent.DelayQueue.take() @bci=57, line=164 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23660: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.DelayQueue.take() @bci=28, line=160 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23651: (state = BLOCKED)
Thread 23650: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=134 (Interpreted frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Interpreted frame)
Thread 23649: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.Object.wait() @bci=2, line=485 (Compiled frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 (Compiled frame)
Thread 23645: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- org.apache.flume.lifecycle.LifecycleController.waitForOneOf(org.apache.flume.lifecycle.LifecycleAware, org.apache.flume.lifecycle.LifecycleState[], long) @bci=109, line=71 (Compiled frame)
- org.apache.flume.lifecycle.LifecycleController.waitForOneOf(org.apache.flume.lifecycle.LifecycleAware, org.apache.flume.lifecycle.LifecycleState[]) @bci=3, line=49 (Interpreted frame)
- org.apache.flume.node.Application.run() @bci=154, line=167 (Interpreted frame)
- org.apache.flume.node.Application.main(java.lang.String[]) @bci=21, line=70 (Interpreted frame)
On Oct 16, 2012, at 11:10 PM, Brock Noland wrote:
> When the agent sees an OOM, can you take a thread dump (kill -3 <pid>) and post it on pastebin? Please be sure to include the heap summary section at the bottom of the dump.
>
> Brock
>
> On Mon, Oct 15, 2012 at 10:08 PM, Yanzhi.liu <40...@qq.com> wrote:
> Hello Brock:
> My flume's configuration is
> agent_foo.sources = tailsource-1
> agent_foo.channels = fileChannel
> agent_foo.sinks = hdfsSink-1
> agent_foo.sources.tailsource-1.type = com.chinacache.cpis.sources.CpisAvroSource
> agent_foo.sources.tailsource-1.channels=fileChannel
> agent_foo.sources.tailsource-1.bind=221.130.18.90
> agent_foo.sources.tailsource-1.port=4545
> #agent_foo.sources.tailsource-1.threads=10
> agent_foo.channels.fileChannel.type = file
> agent_foo.channels.fileChannel.checkpointDir=/home/hadoop/checkpoint
> agent_foo.channels.fileChannel.dataDirs=/data/cache2/data
> agent_foo.channels.fileChannel.maxFileSize=524288000
> #agent_foo.channels.fileChannel.capacity = 10000
> agent_foo.sinks.hdfsSink-1.type = hdfs
> agent_foo.sinks.hdfsSink-1.channel = fileChannel
> agent_foo.sinks.hdfsSink-1.hdfs.path = hdfs://CMN-NJ-2-579:9000/user/hadoop/fc_logs/%{month}/%{day}/%{deviceId}/%{hour}
> #agent_foo.sinks.hdfsSink-1.hdfs.path = hdfs://CMN-NJ-2-579:9000/user/hadoop/fc_logs/%{month}/%{day}/%{hour}/%{deviceId}
> agent_foo.sinks.hdfsSink-1.hdfs.filePrefix = cpisfc-assess.log
> agent_foo.sinks.hdfsSink-1.hdfs.rollInterval=600
> agent_foo.sinks.hdfsSink-1.hdfs.rollCount=0
> agent_foo.sinks.hdfsSink-1.hdfs.rollSize = 506870912
> agent_foo.sinks.hdfsSink-1.hdfs.fileType=CompressedStream
> agent_foo.sinks.hdfsSink-1.hdfs.codeC=gzip
> agent_foo.sinks.hdfsSink-1.hdfs.writeFormat=Text
> agent_foo.sinks.hdfsSink-1.hdfs.batchSize=1
> agent_foo.sinks.hdfsSink-1.serializer=avro_event
> And the java configuration was JAVA_OPTS="-Xms100m -Xmx2048m -Dcom.sun.management.jmxremote";
> Thank you very much!
> My Name :
> Yanzhi Liu
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "Brock Noland"<br...@cloudera.com>;
> 发送时间: 2012年10月16日(星期二) 中午11:01
> 收件人: "user"<us...@flume.apache.org>;
> 主题: Re: About file channel yum error
>
> What is your channel capacity and configured agent heap size?
>
> On Monday, October 15, 2012 at 9:19 PM, Yanzhi.liu wrote:
>
>> Hello everyone:
>> I am using the flume 1.3.0.But the flume clusters was working after two days and the flume was stoped. And the flume.log was only error about the:
>> 14 Oct 2012 15:29:17,018 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.HDFSEventSink.process:448) - process failed
>> java.lang.OutOfMemoryError
>> at java.io.RandomAccessFile.readBytes(Native Method)
>> at java.io.RandomAccessFile.read(RandomAccessFile.java:338)
>> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:397)
>> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377)
>> at org.apache.flume.channel.file.LogFile.readDelimitedBuffer(LogFile.java:486)
>> at org.apache.flume.channel.file.LogFileV3$RandomReader.doGet(LogFileV3.java:258)
>> at org.apache.flume.channel.file.LogFile$RandomReader.get(LogFile.java:298)
>> at org.apache.flume.channel.file.Log.get(Log.java:409)
>> at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:447)
>> at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>> at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:91)
>> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:387)
>> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>> at java.lang.Thread.run(Thread.java:662)
>> So I want to how to avoid the error.
>> Thanks very much!
>> My Name:
>> Yanzhi Liu
>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: 回复： About file channel yum error
Posted by Brock Noland <br...@cloudera.com>.
When the agent sees an OOM, can you take a thread dump (kill -3 <pid>) and
post it on pastebin? Please be sure to include the heap summary section at
the bottom of the dump.
Brock
On Mon, Oct 15, 2012 at 10:08 PM, Yanzhi.liu <40...@qq.com> wrote:
> Hello Brock:
> My flume's configuration is
> agent_foo.sources = tailsource-1
> agent_foo.channels = fileChannel
> agent_foo.sinks = hdfsSink-1
> agent_foo.sources.tailsource-1.type =
> com.chinacache.cpis.sources.CpisAvroSource
> agent_foo.sources.tailsource-1.channels=fileChannel
> agent_foo.sources.tailsource-1.bind=221.130.18.90
> agent_foo.sources.tailsource-1.port=4545
> #agent_foo.sources.tailsource-1.threads=10
> agent_foo.channels.fileChannel.type = file
> agent_foo.channels.fileChannel.checkpointDir=/home/hadoop/checkpoint
> agent_foo.channels.fileChannel.dataDirs=/data/cache2/data
> agent_foo.channels.fileChannel.maxFileSize=524288000
> #agent_foo.channels.fileChannel.capacity = 10000
> agent_foo.sinks.hdfsSink-1.type = hdfs
> agent_foo.sinks.hdfsSink-1.channel = fileChannel
> agent_foo.sinks.hdfsSink-1.hdfs.path =
> hdfs://CMN-NJ-2-579:9000/user/hadoop/fc_logs/%{month}/%{day}/%{deviceId}/%{hour}
> #agent_foo.sinks.hdfsSink-1.hdfs.path =
> hdfs://CMN-NJ-2-579:9000/user/hadoop/fc_logs/%{month}/%{day}/%{hour}/%{deviceId}
> agent_foo.sinks.hdfsSink-1.hdfs.filePrefix = cpisfc-assess.log
> agent_foo.sinks.hdfsSink-1.hdfs.rollInterval=600
> agent_foo.sinks.hdfsSink-1.hdfs.rollCount=0
> agent_foo.sinks.hdfsSink-1.hdfs.rollSize = 506870912
> agent_foo.sinks.hdfsSink-1.hdfs.fileType=CompressedStream
> agent_foo.sinks.hdfsSink-1.hdfs.codeC=gzip
> agent_foo.sinks.hdfsSink-1.hdfs.writeFormat=Text
> agent_foo.sinks.hdfsSink-1.hdfs.batchSize=1
> agent_foo.sinks.hdfsSink-1.serializer=avro_event
> And the java configuration was JAVA_OPTS="-Xms100m -Xmx2048m
> -Dcom.sun.management.jmxremote";
> Thank you very much!
> My Name :
> Yanzhi Liu
> **
>
>
> ------------------ 原始邮件 ------------------
> *发件人:* "Brock Noland"<br...@cloudera.com>;
> *发送时间:* 2012年10月16日(星期二) 中午11:01
> *收件人:* "user"<us...@flume.apache.org>; **
> *主题:* Re: About file channel yum error
>
> What is your channel capacity and configured agent heap size?
>
> On Monday, October 15, 2012 at 9:19 PM, Yanzhi.liu wrote:
>
> Hello everyone:
> I am using the flume 1.3.0.But the flume clusters was working after
> two days and the flume was stoped. And the flume.log was only error about
> the:
> 14 Oct 2012 15:29:17,018 ERROR
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.hdfs.HDFSEventSink.process:448) - process failed
> java.lang.OutOfMemoryError
> at java.io.RandomAccessFile.readBytes(Native Method)
> at java.io.RandomAccessFile.read(RandomAccessFile.java:338)
> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:397)
> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377)
> at
> org.apache.flume.channel.file.LogFile.readDelimitedBuffer(LogFile.java:486)
> at
> org.apache.flume.channel.file.LogFileV3$RandomReader.doGet(LogFileV3.java:258)
> at
> org.apache.flume.channel.file.LogFile$RandomReader.get(LogFile.java:298)
> at org.apache.flume.channel.file.Log.get(Log.java:409)
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:447)
> at
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> at
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:91)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:387)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:662)
> So I want to how to avoid the error.
> Thanks very much!
> My Name:
> Yanzhi Liu
>
>
> **
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/