You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Sharninder <sh...@gmail.com> on 2014/09/21 04:01:13 UTC
Re: flume agent / JVM crashes periodically
Have you increased the JVM heap size or running with the default 20M only?
--
Sharninder
> On 21-Sep-2014, at 3:08 am, terreyshih <te...@gmail.com> wrote:
>
> BTW, no exceptions are thrown. It just started the agent-shutdown-hook.
>
> thx
>
>> On Sep 20, 2014, at 2:33 PM, terreyshih <te...@gmail.com> wrote:
>>
>> Hi,
>>
>> I am having a lot of difficulty trying to debug an issue. The flume server stops working after a few hours and the JVM shuts down. The snippet of the message I got in the log starts with following:
>>
>>
>> 20 Sep 2014 12:24:10,899 DEBUG [New I/O worker #9] (org.hbase.async.RegionClient.decode:1349) - ------------------<< LEAVING DECODE <<------------------ time elapsed: 75us
>> 20 Sep 2014 12:24:19,942 INFO [agent-shutdown-hook] (org.apache.flume.lifecycle.LifecycleSupervisor.stop:79) - Stopping lifecycle supervisor 13
>> 20 Sep 2014 12:24:19,943 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:149) - Component type: CHANNEL, name: myChannelC stopped
>>
>>
>> From there it starts to shut down all the channels and eventually the JVM shuts down.
>>
>> I am running the agent in debug mode and I can see my data coming in correctly for a couple of hours and the snippet shows, flume enters the “LEAVING DECODE” section of the flume debug engine. I am writing the data to HBase sinks by using AsycnHBase.
>>
>> I would appreciate it if someone can help me with this. This is reproducible around every 2 hours.
>>
>> When does the agent-shutdown-hook gets triggered ?
>>
>> thanks
>
Re: flume agent / JVM crashes periodically
Posted by Ashish <pa...@gmail.com>.
Look into the System out/err streams, JVM would dumo Error info there in
case there is one. It may not be part of logs, unless you have redirected
Sys out/err streams to the log file
HTH !
On Sun, Sep 21, 2014 at 9:54 AM, terreyshih <te...@gmail.com> wrote:
> Thanks. So that is the reason why Sharninder thought it might be the JVM
> ?
>
> What do you suggest to look for in this case ?
>
> thanks,
> -Gary
>
>
> On Sep 20, 2014, at 8:47 PM, Hari Shreedharan <hs...@cloudera.com>
> wrote:
>
> The shutdown hook is triggered when the app is shutdown. Since Flume does
> not shutdown by itself, some other external process must have sent it a
> SIGTERM (in a SIGKILL case, the shutdown hook won't execute).
>
> On Sat, Sep 20, 2014 at 7:20 PM, terreyshih <te...@gmail.com> wrote:
>
>> Yes, I have increased to 1G. I have also started the jmx remote
>> monitoring on the jvm which I don’t see any issue.
>>
>> Do you know when the shut down hook kicks in ? (under what conditions ?)
>>
>> thx
>>
>> On Sep 20, 2014, at 7:01 PM, Sharninder <sh...@gmail.com> wrote:
>>
>> Have you increased the JVM heap size or running with the default 20M only?
>>
>> --
>> Sharninder
>>
>> On 21-Sep-2014, at 3:08 am, terreyshih <te...@gmail.com> wrote:
>>
>> BTW, no exceptions are thrown. It just started the agent-shutdown-hook.
>>
>> thx
>>
>> On Sep 20, 2014, at 2:33 PM, terreyshih <te...@gmail.com> wrote:
>>
>> Hi,
>>
>> I am having a lot of difficulty trying to debug an issue. The flume
>> server stops working after a few hours and the JVM shuts down. The snippet
>> of the message I got in the log starts with following:
>>
>>
>> 20 Sep 2014 12:24:10,899 DEBUG [New I/O worker #9]
>> (org.hbase.async.RegionClient.decode:1349) - ------------------<< LEAVING
>> DECODE <<------------------ time elapsed: 75us
>> 20 Sep 2014 12:24:19,942 INFO [agent-shutdown-hook]
>> (org.apache.flume.lifecycle.LifecycleSupervisor.stop:79) - Stopping
>> lifecycle supervisor 13
>> 20 Sep 2014 12:24:19,943 INFO [agent-shutdown-hook]
>> (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:149) -
>> Component type: CHANNEL, name: myChannelC stopped
>>
>>
>> From there it starts to shut down all the channels and eventually the JVM
>> shuts down.
>>
>> I am running the agent in debug mode and I can see my data coming in
>> correctly for a couple of hours and the snippet shows, flume enters the
>> “LEAVING DECODE” section of the flume debug engine. I am writing the data
>> to HBase sinks by using AsycnHBase.
>>
>> I would appreciate it if someone can help me with this. This is
>> reproducible around every 2 hours.
>>
>> When does the agent-shutdown-hook gets triggered ?
>>
>> thanks
>>
>>
>>
>>
>
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: flume agent / JVM crashes periodically
Posted by terreyshih <te...@gmail.com>.
Thanks. So that is the reason why Sharninder thought it might be the JVM ?
What do you suggest to look for in this case ?
thanks,
-Gary
On Sep 20, 2014, at 8:47 PM, Hari Shreedharan <hs...@cloudera.com> wrote:
> The shutdown hook is triggered when the app is shutdown. Since Flume does not shutdown by itself, some other external process must have sent it a SIGTERM (in a SIGKILL case, the shutdown hook won't execute).
>
> On Sat, Sep 20, 2014 at 7:20 PM, terreyshih <te...@gmail.com> wrote:
> Yes, I have increased to 1G. I have also started the jmx remote monitoring on the jvm which I don’t see any issue.
>
> Do you know when the shut down hook kicks in ? (under what conditions ?)
>
> thx
>
> On Sep 20, 2014, at 7:01 PM, Sharninder <sh...@gmail.com> wrote:
>
>> Have you increased the JVM heap size or running with the default 20M only?
>>
>> --
>> Sharninder
>>
>> On 21-Sep-2014, at 3:08 am, terreyshih <te...@gmail.com> wrote:
>>
>>> BTW, no exceptions are thrown. It just started the agent-shutdown-hook.
>>>
>>> thx
>>>
>>> On Sep 20, 2014, at 2:33 PM, terreyshih <te...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am having a lot of difficulty trying to debug an issue. The flume server stops working after a few hours and the JVM shuts down. The snippet of the message I got in the log starts with following:
>>>>
>>>>
>>>> 20 Sep 2014 12:24:10,899 DEBUG [New I/O worker #9] (org.hbase.async.RegionClient.decode:1349) - ------------------<< LEAVING DECODE <<------------------ time elapsed: 75us
>>>> 20 Sep 2014 12:24:19,942 INFO [agent-shutdown-hook] (org.apache.flume.lifecycle.LifecycleSupervisor.stop:79) - Stopping lifecycle supervisor 13
>>>> 20 Sep 2014 12:24:19,943 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:149) - Component type: CHANNEL, name: myChannelC stopped
>>>>
>>>>
>>>> From there it starts to shut down all the channels and eventually the JVM shuts down.
>>>>
>>>> I am running the agent in debug mode and I can see my data coming in correctly for a couple of hours and the snippet shows, flume enters the “LEAVING DECODE” section of the flume debug engine. I am writing the data to HBase sinks by using AsycnHBase.
>>>>
>>>> I would appreciate it if someone can help me with this. This is reproducible around every 2 hours.
>>>>
>>>> When does the agent-shutdown-hook gets triggered ?
>>>>
>>>> thanks
>>>
>
>
Re: flume agent / JVM crashes periodically
Posted by Hari Shreedharan <hs...@cloudera.com>.
The shutdown hook is triggered when the app is shutdown. Since Flume does
not shutdown by itself, some other external process must have sent it a
SIGTERM (in a SIGKILL case, the shutdown hook won't execute).
On Sat, Sep 20, 2014 at 7:20 PM, terreyshih <te...@gmail.com> wrote:
> Yes, I have increased to 1G. I have also started the jmx remote
> monitoring on the jvm which I don’t see any issue.
>
> Do you know when the shut down hook kicks in ? (under what conditions ?)
>
> thx
>
> On Sep 20, 2014, at 7:01 PM, Sharninder <sh...@gmail.com> wrote:
>
> Have you increased the JVM heap size or running with the default 20M only?
>
> --
> Sharninder
>
> On 21-Sep-2014, at 3:08 am, terreyshih <te...@gmail.com> wrote:
>
> BTW, no exceptions are thrown. It just started the agent-shutdown-hook.
>
> thx
>
> On Sep 20, 2014, at 2:33 PM, terreyshih <te...@gmail.com> wrote:
>
> Hi,
>
> I am having a lot of difficulty trying to debug an issue. The flume
> server stops working after a few hours and the JVM shuts down. The snippet
> of the message I got in the log starts with following:
>
>
> 20 Sep 2014 12:24:10,899 DEBUG [New I/O worker #9]
> (org.hbase.async.RegionClient.decode:1349) - ------------------<< LEAVING
> DECODE <<------------------ time elapsed: 75us
> 20 Sep 2014 12:24:19,942 INFO [agent-shutdown-hook]
> (org.apache.flume.lifecycle.LifecycleSupervisor.stop:79) - Stopping
> lifecycle supervisor 13
> 20 Sep 2014 12:24:19,943 INFO [agent-shutdown-hook]
> (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:149) -
> Component type: CHANNEL, name: myChannelC stopped
>
>
> From there it starts to shut down all the channels and eventually the JVM
> shuts down.
>
> I am running the agent in debug mode and I can see my data coming in
> correctly for a couple of hours and the snippet shows, flume enters the
> “LEAVING DECODE” section of the flume debug engine. I am writing the data
> to HBase sinks by using AsycnHBase.
>
> I would appreciate it if someone can help me with this. This is
> reproducible around every 2 hours.
>
> When does the agent-shutdown-hook gets triggered ?
>
> thanks
>
>
>
>
Re: flume agent / JVM crashes periodically
Posted by terreyshih <te...@gmail.com>.
Yes, I have increased to 1G. I have also started the jmx remote monitoring on the jvm which I don’t see any issue.
Do you know when the shut down hook kicks in ? (under what conditions ?)
thx
On Sep 20, 2014, at 7:01 PM, Sharninder <sh...@gmail.com> wrote:
> Have you increased the JVM heap size or running with the default 20M only?
>
> --
> Sharninder
>
> On 21-Sep-2014, at 3:08 am, terreyshih <te...@gmail.com> wrote:
>
>> BTW, no exceptions are thrown. It just started the agent-shutdown-hook.
>>
>> thx
>>
>> On Sep 20, 2014, at 2:33 PM, terreyshih <te...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am having a lot of difficulty trying to debug an issue. The flume server stops working after a few hours and the JVM shuts down. The snippet of the message I got in the log starts with following:
>>>
>>>
>>> 20 Sep 2014 12:24:10,899 DEBUG [New I/O worker #9] (org.hbase.async.RegionClient.decode:1349) - ------------------<< LEAVING DECODE <<------------------ time elapsed: 75us
>>> 20 Sep 2014 12:24:19,942 INFO [agent-shutdown-hook] (org.apache.flume.lifecycle.LifecycleSupervisor.stop:79) - Stopping lifecycle supervisor 13
>>> 20 Sep 2014 12:24:19,943 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:149) - Component type: CHANNEL, name: myChannelC stopped
>>>
>>>
>>> From there it starts to shut down all the channels and eventually the JVM shuts down.
>>>
>>> I am running the agent in debug mode and I can see my data coming in correctly for a couple of hours and the snippet shows, flume enters the “LEAVING DECODE” section of the flume debug engine. I am writing the data to HBase sinks by using AsycnHBase.
>>>
>>> I would appreciate it if someone can help me with this. This is reproducible around every 2 hours.
>>>
>>> When does the agent-shutdown-hook gets triggered ?
>>>
>>> thanks
>>