You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Lenin Raj <em...@gmail.com> on 2013/06/01 09:47:01 UTC
Flume service stopped automatically
Hello,
I have a flume service which pulls twitter data and sinks to HDFS. I
started it last night at 8 PM. It was going well. When I checked the status
today morning, it seems flume service stopped at around 11:15 PM.
I could not find any valid reason in the log file. Flume is doing "Flume
node stopping - TwitterAgent" automatically without throwing any error.
31 May 2013 23:13:32,226 INFO [hdfs-HDFS-call-runner-7]
(org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating
hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414624.tmp
31 May 2013 23:14:02,247 INFO [hdfs-HDFS-roll-timer-0]
(org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427) - Renaming
hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414624.tmp
to
hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414624
31 May 2013 23:14:26,260 INFO [hdfs-HDFS-call-runner-1]
(org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating
hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414625.tmp
31 May 2013 23:14:56,288 INFO [hdfs-HDFS-roll-timer-0]
(org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427) - Renaming
hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414625.tmp
to
hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414625
31 May 2013 23:15:02,317 INFO [hdfs-HDFS-call-runner-5]
(org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating
hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414626.tmp
31 May 2013 23:15:32,340 INFO [hdfs-HDFS-roll-timer-0]
(org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427) - Renaming
hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414626.tmp
to
hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414626
31 May 2013 23:15:43,632 INFO [hdfs-HDFS-call-runner-9]
(org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating
hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414627.tmp
31 May 2013 23:15:56,604 INFO [node-shutdownHook]
(org.apache.flume.node.FlumeNode.stop:67) - Flume node stopping -
TwitterAgent
31 May 2013 23:15:56,604 INFO [node-shutdownHook]
(org.apache.flume.lifecycle.LifecycleSupervisor.stop:78) - Stopping
lifecycle supervisor 7
31 May 2013 23:15:56,606 INFO [node-shutdownHook]
(org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stop:215) -
Node manager stopping
31 May 2013 23:15:56,606 INFO [node-shutdownHook]
(org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:68)
- Shutting down configuration: {
sourceRunners:{Twitter=EventDrivenSourceRunner: {
source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:START} }}
sinkRunners:{HDFS=SinkRunner: {
policy:org.apache.flume.sink.DefaultSinkProcessor@1d80e6d counterGroup:{
name:null counters:{runner.backoffs.consecutive=2, runner.backoffs=1252}
} }} channels:{MemChannel=org.apache.flume.channel.MemoryChannel{name:
MemChannel}} }
31 May 2013 23:15:56,607 INFO [node-shutdownHook]
(org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:72)
- Stopping Source Twitter
31 May 2013 23:15:56,607 INFO [node-shutdownHook]
(org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:156) -
Stopping component: EventDrivenSourceRunner: {
source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:START} }
31 May 2013 23:15:56,682 WARN [Twitter Stream consumer-1[Disposing
thread]] (twitter4j.internal.logging.SLF4JLogger.warn:99) - Stream already
closed.
31 May 2013 23:15:56,682 INFO [node-shutdownHook]
(org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:82)
- Stopping Sink HDFS
31 May 2013 23:15:56,683 INFO [node-shutdownHook]
(org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:156) -
Stopping component: SinkRunner: {
policy:org.apache.flume.sink.DefaultSinkProcessor@1d80e6d counterGroup:{
name:null counters:{runner.backoffs.consecutive=2, runner.backoffs=1252} } }
31 May 2013 23:15:56,754 WARN
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout:372) -
Unexpected Exception null
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325)
at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257)
at java.util.concurrent.FutureTask.get(FutureTask.java:119)
at
org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:345)
at
org.apache.flume.sink.hdfs.HDFSEventSink.flush(HDFSEventSink.java:741)
at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:443)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:679)
31 May 2013 23:15:56,797 ERROR
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.flume.sink.hdfs.HDFSEventSink.process:460) - process failed
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325)
at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257)
at java.util.concurrent.FutureTask.get(FutureTask.java:119)
at
org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:345)
at
org.apache.flume.sink.hdfs.HDFSEventSink.flush(HDFSEventSink.java:741)
at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:443)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:679)
31 May 2013 23:15:56,797 ERROR
[SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver
event. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.InterruptedException
at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:464)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325)
at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257)
at java.util.concurrent.FutureTask.get(FutureTask.java:119)
at
org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:345)
at
org.apache.flume.sink.hdfs.HDFSEventSink.flush(HDFSEventSink.java:741)
at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:443)
... 3 more
31 May 2013 23:16:01,798 INFO [node-shutdownHook]
(org.apache.flume.sink.hdfs.HDFSEventSink.stop:475) - Closing
hdfs://localhost:54310/user/flume/tweets/2013/05/31/20//FlumeData
31 May 2013 23:16:01,811 INFO [hdfs-HDFS-call-runner-4]
(org.apache.flume.sink.hdfs.BucketWriter.doClose:283) - HDFSWriter is
already closed:
hdfs://localhost:54310/user/flume/tweets/2013/05/31/20//FlumeData.1370013399555.tmp
31 May 2013 23:16:01,812 INFO [node-shutdownHook]
(org.apache.flume.sink.hdfs.HDFSEventSink.stop:475) - Closing
hdfs://localhost:54310/user/flume/tweets/2013/05/31/21//FlumeData
Any reason Flume stopped the service?
Thanks,
Lenin
Re: Flume service stopped automatically
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
On Linux:
dmesg
or
/var/log/messages
- Alex
On Jun 1, 2013, at 4:49 PM, Lenin Raj <em...@gmail.com> wrote:
> Hi Brock,
>
> Which log file should I check under /var/log?
>
> Following are the list of files under /var/log modified from yesterdat:
>
> -rw-r----- 1 syslog adm 1227 2013-05-31 06:25 syslog.2.gz
> -rw-r----- 1 mysql adm 20 2013-05-31 06:25 mysql.log.1.gz
> -rw-rw-r-- 1 root utmp 40704 2013-05-31 23:15 wtmp.1
> -rw-r----- 1 syslog adm 14296 2013-06-01 06:25 syslog.1
> -rw-r--r-- 1 root root 0 2013-06-01 06:25 dpkg.log
> drwxr-xr-x 2 root root 4096 2013-06-01 06:25 apt
> -rw-r--r-- 1 root root 0 2013-06-01 06:25 alternatives.log
> -rw-r----- 1 mysql adm 0 2013-06-01 06:25 mysql.log
> drwxr-s--- 2 Debian-exim adm 4096 2013-06-01 06:25 exim4
> -rw-rw---- 1 root utmp 0 2013-06-01 06:25 btmp
> -rw-r----- 1 syslog adm 119666 2013-06-01 18:03 kern.log
> -rw-r----- 1 syslog adm 8845 2013-06-01 21:09 syslog
> -rw-r----- 1 syslog adm 4713281 2013-06-01 21:14 auth.log
> -rw-rw-r-- 1 root utmp 3456 2013-06-01 21:14 wtmp
> -rw-r--r-- 1 root root 293752 2013-06-01 21:14 lastlog
>
>
>
> Thanks,
> Lenin
>
>
> On Sat, Jun 1, 2013 at 9:12 PM, Brock Noland <br...@cloudera.com> wrote:
>
>> Weird... It could have been the linux oom killer? You'd see something in
>> /var/log/message if that was the case.
>>
>>
>> On Sat, Jun 1, 2013 at 2:47 AM, Lenin Raj <em...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I have a flume service which pulls twitter data and sinks to HDFS. I
>>> started it last night at 8 PM. It was going well. When I checked the status
>>> today morning, it seems flume service stopped at around 11:15 PM.
>>>
>>> I could not find any valid reason in the log file. Flume is doing "Flume
>>> node stopping - TwitterAgent" automatically without throwing any error.
>>>
>>>
>>> 31 May 2013 23:13:32,226 INFO [hdfs-HDFS-call-runner-7]
>>> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414624.tmp
>>> 31 May 2013 23:14:02,247 INFO [hdfs-HDFS-roll-timer-0]
>>> (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427) - Renaming
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414624.tmp
>>> to
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414624
>>> 31 May 2013 23:14:26,260 INFO [hdfs-HDFS-call-runner-1]
>>> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414625.tmp
>>> 31 May 2013 23:14:56,288 INFO [hdfs-HDFS-roll-timer-0]
>>> (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427) - Renaming
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414625.tmp
>>> to
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414625
>>> 31 May 2013 23:15:02,317 INFO [hdfs-HDFS-call-runner-5]
>>> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414626.tmp
>>> 31 May 2013 23:15:32,340 INFO [hdfs-HDFS-roll-timer-0]
>>> (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427) - Renaming
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414626.tmp
>>> to
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414626
>>> 31 May 2013 23:15:43,632 INFO [hdfs-HDFS-call-runner-9]
>>> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414627.tmp
>>> 31 May 2013 23:15:56,604 INFO [node-shutdownHook]
>>> (org.apache.flume.node.FlumeNode.stop:67) - Flume node stopping -
>>> TwitterAgent
>>> 31 May 2013 23:15:56,604 INFO [node-shutdownHook]
>>> (org.apache.flume.lifecycle.LifecycleSupervisor.stop:78) - Stopping
>>> lifecycle supervisor 7
>>> 31 May 2013 23:15:56,606 INFO [node-shutdownHook]
>>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stop:215) -
>>> Node manager stopping
>>> 31 May 2013 23:15:56,606 INFO [node-shutdownHook]
>>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:68)
>>> - Shutting down configuration: {
>>> sourceRunners:{Twitter=EventDrivenSourceRunner: {
>>> source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:START} }}
>>> sinkRunners:{HDFS=SinkRunner: {
>>> policy:org.apache.flume.sink.DefaultSinkProcessor@1d80e6d counterGroup:{
>>> name:null counters:{runner.backoffs.consecutive=2, runner.backoffs=1252}
>>> } }} channels:{MemChannel=org.apache.flume.channel.MemoryChannel{name:
>>> MemChannel}} }
>>> 31 May 2013 23:15:56,607 INFO [node-shutdownHook]
>>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:72)
>>> - Stopping Source Twitter
>>> 31 May 2013 23:15:56,607 INFO [node-shutdownHook]
>>> (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:156) -
>>> Stopping component: EventDrivenSourceRunner: {
>>> source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:START} }
>>> 31 May 2013 23:15:56,682 WARN [Twitter Stream consumer-1[Disposing
>>> thread]] (twitter4j.internal.logging.SLF4JLogger.warn:99) - Stream already
>>> closed.
>>> 31 May 2013 23:15:56,682 INFO [node-shutdownHook]
>>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:82)
>>> - Stopping Sink HDFS
>>> 31 May 2013 23:15:56,683 INFO [node-shutdownHook]
>>> (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:156) -
>>> Stopping component: SinkRunner: {
>>> policy:org.apache.flume.sink.DefaultSinkProcessor@1d80e6d counterGroup:{
>>> name:null counters:{runner.backoffs.consecutive=2, runner.backoffs=1252} } }
>>> 31 May 2013 23:15:56,754 WARN
>>> [SinkRunner-PollingRunner-DefaultSinkProcessor]
>>> (org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout:372) -
>>> Unexpected Exception null
>>> java.lang.InterruptedException
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325)
>>> at
>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257)
>>> at java.util.concurrent.FutureTask.get(FutureTask.java:119)
>>> at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:345)
>>> at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.flush(HDFSEventSink.java:741)
>>> at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:443)
>>> at
>>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>> at
>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>> at java.lang.Thread.run(Thread.java:679)
>>> 31 May 2013 23:15:56,797 ERROR
>>> [SinkRunner-PollingRunner-DefaultSinkProcessor]
>>> (org.apache.flume.sink.hdfs.HDFSEventSink.process:460) - process failed
>>> java.lang.InterruptedException
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325)
>>> at
>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257)
>>> at java.util.concurrent.FutureTask.get(FutureTask.java:119)
>>> at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:345)
>>> at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.flush(HDFSEventSink.java:741)
>>> at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:443)
>>> at
>>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>> at
>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>> at java.lang.Thread.run(Thread.java:679)
>>> 31 May 2013 23:15:56,797 ERROR
>>> [SinkRunner-PollingRunner-DefaultSinkProcessor]
>>> (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver
>>> event. Exception follows.
>>> org.apache.flume.EventDeliveryException: java.lang.InterruptedException
>>> at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:464)
>>> at
>>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>> at
>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>> at java.lang.Thread.run(Thread.java:679)
>>> Caused by: java.lang.InterruptedException
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325)
>>> at
>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257)
>>> at java.util.concurrent.FutureTask.get(FutureTask.java:119)
>>> at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:345)
>>> at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.flush(HDFSEventSink.java:741)
>>> at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:443)
>>> ... 3 more
>>> 31 May 2013 23:16:01,798 INFO [node-shutdownHook]
>>> (org.apache.flume.sink.hdfs.HDFSEventSink.stop:475) - Closing
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/20//FlumeData
>>> 31 May 2013 23:16:01,811 INFO [hdfs-HDFS-call-runner-4]
>>> (org.apache.flume.sink.hdfs.BucketWriter.doClose:283) - HDFSWriter is
>>> already closed:
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/20//FlumeData.1370013399555.tmp
>>> 31 May 2013 23:16:01,812 INFO [node-shutdownHook]
>>> (org.apache.flume.sink.hdfs.HDFSEventSink.stop:475) - Closing
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/21//FlumeData
>>>
>>> Any reason Flume stopped the service?
>>>
>>> Thanks,
>>> Lenin
>>>
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>>
Re: Flume service stopped automatically
Posted by Lenin Raj <em...@gmail.com>.
Hi Brock,
Which log file should I check under /var/log?
Following are the list of files under /var/log modified from yesterdat:
-rw-r----- 1 syslog adm 1227 2013-05-31 06:25 syslog.2.gz
-rw-r----- 1 mysql adm 20 2013-05-31 06:25 mysql.log.1.gz
-rw-rw-r-- 1 root utmp 40704 2013-05-31 23:15 wtmp.1
-rw-r----- 1 syslog adm 14296 2013-06-01 06:25 syslog.1
-rw-r--r-- 1 root root 0 2013-06-01 06:25 dpkg.log
drwxr-xr-x 2 root root 4096 2013-06-01 06:25 apt
-rw-r--r-- 1 root root 0 2013-06-01 06:25 alternatives.log
-rw-r----- 1 mysql adm 0 2013-06-01 06:25 mysql.log
drwxr-s--- 2 Debian-exim adm 4096 2013-06-01 06:25 exim4
-rw-rw---- 1 root utmp 0 2013-06-01 06:25 btmp
-rw-r----- 1 syslog adm 119666 2013-06-01 18:03 kern.log
-rw-r----- 1 syslog adm 8845 2013-06-01 21:09 syslog
-rw-r----- 1 syslog adm 4713281 2013-06-01 21:14 auth.log
-rw-rw-r-- 1 root utmp 3456 2013-06-01 21:14 wtmp
-rw-r--r-- 1 root root 293752 2013-06-01 21:14 lastlog
Thanks,
Lenin
On Sat, Jun 1, 2013 at 9:12 PM, Brock Noland <br...@cloudera.com> wrote:
> Weird... It could have been the linux oom killer? You'd see something in
> /var/log/message if that was the case.
>
>
> On Sat, Jun 1, 2013 at 2:47 AM, Lenin Raj <em...@gmail.com> wrote:
>
>> Hello,
>>
>> I have a flume service which pulls twitter data and sinks to HDFS. I
>> started it last night at 8 PM. It was going well. When I checked the status
>> today morning, it seems flume service stopped at around 11:15 PM.
>>
>> I could not find any valid reason in the log file. Flume is doing "Flume
>> node stopping - TwitterAgent" automatically without throwing any error.
>>
>>
>> 31 May 2013 23:13:32,226 INFO [hdfs-HDFS-call-runner-7]
>> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating
>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414624.tmp
>> 31 May 2013 23:14:02,247 INFO [hdfs-HDFS-roll-timer-0]
>> (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427) - Renaming
>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414624.tmp
>> to
>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414624
>> 31 May 2013 23:14:26,260 INFO [hdfs-HDFS-call-runner-1]
>> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating
>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414625.tmp
>> 31 May 2013 23:14:56,288 INFO [hdfs-HDFS-roll-timer-0]
>> (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427) - Renaming
>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414625.tmp
>> to
>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414625
>> 31 May 2013 23:15:02,317 INFO [hdfs-HDFS-call-runner-5]
>> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating
>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414626.tmp
>> 31 May 2013 23:15:32,340 INFO [hdfs-HDFS-roll-timer-0]
>> (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427) - Renaming
>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414626.tmp
>> to
>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414626
>> 31 May 2013 23:15:43,632 INFO [hdfs-HDFS-call-runner-9]
>> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating
>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414627.tmp
>> 31 May 2013 23:15:56,604 INFO [node-shutdownHook]
>> (org.apache.flume.node.FlumeNode.stop:67) - Flume node stopping -
>> TwitterAgent
>> 31 May 2013 23:15:56,604 INFO [node-shutdownHook]
>> (org.apache.flume.lifecycle.LifecycleSupervisor.stop:78) - Stopping
>> lifecycle supervisor 7
>> 31 May 2013 23:15:56,606 INFO [node-shutdownHook]
>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stop:215) -
>> Node manager stopping
>> 31 May 2013 23:15:56,606 INFO [node-shutdownHook]
>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:68)
>> - Shutting down configuration: {
>> sourceRunners:{Twitter=EventDrivenSourceRunner: {
>> source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:START} }}
>> sinkRunners:{HDFS=SinkRunner: {
>> policy:org.apache.flume.sink.DefaultSinkProcessor@1d80e6d counterGroup:{
>> name:null counters:{runner.backoffs.consecutive=2, runner.backoffs=1252}
>> } }} channels:{MemChannel=org.apache.flume.channel.MemoryChannel{name:
>> MemChannel}} }
>> 31 May 2013 23:15:56,607 INFO [node-shutdownHook]
>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:72)
>> - Stopping Source Twitter
>> 31 May 2013 23:15:56,607 INFO [node-shutdownHook]
>> (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:156) -
>> Stopping component: EventDrivenSourceRunner: {
>> source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:START} }
>> 31 May 2013 23:15:56,682 WARN [Twitter Stream consumer-1[Disposing
>> thread]] (twitter4j.internal.logging.SLF4JLogger.warn:99) - Stream already
>> closed.
>> 31 May 2013 23:15:56,682 INFO [node-shutdownHook]
>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:82)
>> - Stopping Sink HDFS
>> 31 May 2013 23:15:56,683 INFO [node-shutdownHook]
>> (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:156) -
>> Stopping component: SinkRunner: {
>> policy:org.apache.flume.sink.DefaultSinkProcessor@1d80e6d counterGroup:{
>> name:null counters:{runner.backoffs.consecutive=2, runner.backoffs=1252} } }
>> 31 May 2013 23:15:56,754 WARN
>> [SinkRunner-PollingRunner-DefaultSinkProcessor]
>> (org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout:372) -
>> Unexpected Exception null
>> java.lang.InterruptedException
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325)
>> at
>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257)
>> at java.util.concurrent.FutureTask.get(FutureTask.java:119)
>> at
>> org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:345)
>> at
>> org.apache.flume.sink.hdfs.HDFSEventSink.flush(HDFSEventSink.java:741)
>> at
>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:443)
>> at
>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>> at
>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>> at java.lang.Thread.run(Thread.java:679)
>> 31 May 2013 23:15:56,797 ERROR
>> [SinkRunner-PollingRunner-DefaultSinkProcessor]
>> (org.apache.flume.sink.hdfs.HDFSEventSink.process:460) - process failed
>> java.lang.InterruptedException
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325)
>> at
>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257)
>> at java.util.concurrent.FutureTask.get(FutureTask.java:119)
>> at
>> org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:345)
>> at
>> org.apache.flume.sink.hdfs.HDFSEventSink.flush(HDFSEventSink.java:741)
>> at
>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:443)
>> at
>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>> at
>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>> at java.lang.Thread.run(Thread.java:679)
>> 31 May 2013 23:15:56,797 ERROR
>> [SinkRunner-PollingRunner-DefaultSinkProcessor]
>> (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver
>> event. Exception follows.
>> org.apache.flume.EventDeliveryException: java.lang.InterruptedException
>> at
>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:464)
>> at
>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>> at
>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>> at java.lang.Thread.run(Thread.java:679)
>> Caused by: java.lang.InterruptedException
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325)
>> at
>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257)
>> at java.util.concurrent.FutureTask.get(FutureTask.java:119)
>> at
>> org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:345)
>> at
>> org.apache.flume.sink.hdfs.HDFSEventSink.flush(HDFSEventSink.java:741)
>> at
>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:443)
>> ... 3 more
>> 31 May 2013 23:16:01,798 INFO [node-shutdownHook]
>> (org.apache.flume.sink.hdfs.HDFSEventSink.stop:475) - Closing
>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/20//FlumeData
>> 31 May 2013 23:16:01,811 INFO [hdfs-HDFS-call-runner-4]
>> (org.apache.flume.sink.hdfs.BucketWriter.doClose:283) - HDFSWriter is
>> already closed:
>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/20//FlumeData.1370013399555.tmp
>> 31 May 2013 23:16:01,812 INFO [node-shutdownHook]
>> (org.apache.flume.sink.hdfs.HDFSEventSink.stop:475) - Closing
>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/21//FlumeData
>>
>> Any reason Flume stopped the service?
>>
>> Thanks,
>> Lenin
>>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>
Re: Flume service stopped automatically
Posted by Brock Noland <br...@cloudera.com>.
Weird... It could have been the linux oom killer? You'd see something in
/var/log/message if that was the case.
On Sat, Jun 1, 2013 at 2:47 AM, Lenin Raj <em...@gmail.com> wrote:
> Hello,
>
> I have a flume service which pulls twitter data and sinks to HDFS. I
> started it last night at 8 PM. It was going well. When I checked the status
> today morning, it seems flume service stopped at around 11:15 PM.
>
> I could not find any valid reason in the log file. Flume is doing "Flume
> node stopping - TwitterAgent" automatically without throwing any error.
>
>
> 31 May 2013 23:13:32,226 INFO [hdfs-HDFS-call-runner-7]
> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414624.tmp
> 31 May 2013 23:14:02,247 INFO [hdfs-HDFS-roll-timer-0]
> (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427) - Renaming
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414624.tmp
> to
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414624
> 31 May 2013 23:14:26,260 INFO [hdfs-HDFS-call-runner-1]
> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414625.tmp
> 31 May 2013 23:14:56,288 INFO [hdfs-HDFS-roll-timer-0]
> (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427) - Renaming
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414625.tmp
> to
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414625
> 31 May 2013 23:15:02,317 INFO [hdfs-HDFS-call-runner-5]
> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414626.tmp
> 31 May 2013 23:15:32,340 INFO [hdfs-HDFS-roll-timer-0]
> (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427) - Renaming
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414626.tmp
> to
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414626
> 31 May 2013 23:15:43,632 INFO [hdfs-HDFS-call-runner-9]
> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208) - Creating
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414627.tmp
> 31 May 2013 23:15:56,604 INFO [node-shutdownHook]
> (org.apache.flume.node.FlumeNode.stop:67) - Flume node stopping -
> TwitterAgent
> 31 May 2013 23:15:56,604 INFO [node-shutdownHook]
> (org.apache.flume.lifecycle.LifecycleSupervisor.stop:78) - Stopping
> lifecycle supervisor 7
> 31 May 2013 23:15:56,606 INFO [node-shutdownHook]
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stop:215) -
> Node manager stopping
> 31 May 2013 23:15:56,606 INFO [node-shutdownHook]
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:68)
> - Shutting down configuration: {
> sourceRunners:{Twitter=EventDrivenSourceRunner: {
> source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:START} }}
> sinkRunners:{HDFS=SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@1d80e6d counterGroup:{
> name:null counters:{runner.backoffs.consecutive=2, runner.backoffs=1252}
> } }} channels:{MemChannel=org.apache.flume.channel.MemoryChannel{name:
> MemChannel}} }
> 31 May 2013 23:15:56,607 INFO [node-shutdownHook]
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:72)
> - Stopping Source Twitter
> 31 May 2013 23:15:56,607 INFO [node-shutdownHook]
> (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:156) -
> Stopping component: EventDrivenSourceRunner: {
> source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:START} }
> 31 May 2013 23:15:56,682 WARN [Twitter Stream consumer-1[Disposing
> thread]] (twitter4j.internal.logging.SLF4JLogger.warn:99) - Stream already
> closed.
> 31 May 2013 23:15:56,682 INFO [node-shutdownHook]
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:82)
> - Stopping Sink HDFS
> 31 May 2013 23:15:56,683 INFO [node-shutdownHook]
> (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:156) -
> Stopping component: SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@1d80e6d counterGroup:{
> name:null counters:{runner.backoffs.consecutive=2, runner.backoffs=1252} } }
> 31 May 2013 23:15:56,754 WARN
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout:372) -
> Unexpected Exception null
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325)
> at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257)
> at java.util.concurrent.FutureTask.get(FutureTask.java:119)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:345)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.flush(HDFSEventSink.java:741)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:443)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:679)
> 31 May 2013 23:15:56,797 ERROR
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.hdfs.HDFSEventSink.process:460) - process failed
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325)
> at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257)
> at java.util.concurrent.FutureTask.get(FutureTask.java:119)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:345)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.flush(HDFSEventSink.java:741)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:443)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:679)
> 31 May 2013 23:15:56,797 ERROR
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver
> event. Exception follows.
> org.apache.flume.EventDeliveryException: java.lang.InterruptedException
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:464)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325)
> at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257)
> at java.util.concurrent.FutureTask.get(FutureTask.java:119)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:345)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.flush(HDFSEventSink.java:741)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:443)
> ... 3 more
> 31 May 2013 23:16:01,798 INFO [node-shutdownHook]
> (org.apache.flume.sink.hdfs.HDFSEventSink.stop:475) - Closing
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/20//FlumeData
> 31 May 2013 23:16:01,811 INFO [hdfs-HDFS-call-runner-4]
> (org.apache.flume.sink.hdfs.BucketWriter.doClose:283) - HDFSWriter is
> already closed:
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/20//FlumeData.1370013399555.tmp
> 31 May 2013 23:16:01,812 INFO [node-shutdownHook]
> (org.apache.flume.sink.hdfs.HDFSEventSink.stop:475) - Closing
> hdfs://localhost:54310/user/flume/tweets/2013/05/31/21//FlumeData
>
> Any reason Flume stopped the service?
>
> Thanks,
> Lenin
>
--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org