You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Shengjie Min <ke...@gmail.com> on 2012/12/19 12:34:59 UTC
flume master down, flume node still has multiple orphan processes
Hi,
I noticed that when my flume master is down for a quite while. My flume
agent is retrying to connect to the master, i can tell that from the flume
logs as below;
2012-12-19 11:11:42,126 WARN com.cloudera.flume.agent.MultiMasterRPC: Could
not connect to any master nodes (tried 1: [flume-master-01:35872])
2012-12-19 11:11:52,140 WARN com.cloudera.flume.agent.MultiMasterRPC: Could
not connect to any master nodes (tried 1: [flume-master-01:35872])
2012-12-19 11:12:02,152 WARN com.cloudera.flume.agent.MultiMasterRPC: Could
not connect to any master nodes (tried 1: [flume-master-01:35872])
2012-12-19 11:12:12,162 WARN com.cloudera.flume.agent.MultiMasterRPC: Could
not connect to any master nodes (tried 1: [flume-master-01:35872])
from "top", I've seen a lot of java processes owned by flume. followed my
instinct, I stopped flume agent, after a while, I run "top" on it, I still
have a lot of processes owned by flume user.
12735 flume 20 0 1487m 26m 128 S 0.3 1.3 0:05.41 java
13828 flume 20 0 1487m 24 24 S 0.3 0.0 8:11.92 java
14442 flume 20 0 1487m 0 0 S 0.3 0.0 8:06.81 java
15033 flume 20 0 1487m 26m 164 S 0.3 1.3 0:04.29 java
15453 flume 20 0 1487m 4 4 S 0.3 0.0 8:07.55 java
15964 flume 20 0 1487m 27m 164 S 0.3 1.4 0:04.09 java
16098 flume 20 0 1487m 0 0 S 0.3 0.0 8:12.67 java
......................
Why so many processes are generated here and they don't go away even when
flume is stopped.
--
All the best,
Shengjie Min
Re: flume master down, flume node still has multiple orphan processes
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
You have to kill -9 them, when you let run a jstack on a JVM pid you'll see WAITING threads.
Best,
Alex
On Dec 19, 2012, at 12:34 PM, Shengjie Min <ke...@gmail.com> wrote:
> Hi,
>
> I noticed that when my flume master is down for a quite while. My flume
> agent is retrying to connect to the master, i can tell that from the flume
> logs as below;
>
> 2012-12-19 11:11:42,126 WARN com.cloudera.flume.agent.MultiMasterRPC: Could
> not connect to any master nodes (tried 1: [flume-master-01:35872])
> 2012-12-19 11:11:52,140 WARN com.cloudera.flume.agent.MultiMasterRPC: Could
> not connect to any master nodes (tried 1: [flume-master-01:35872])
> 2012-12-19 11:12:02,152 WARN com.cloudera.flume.agent.MultiMasterRPC: Could
> not connect to any master nodes (tried 1: [flume-master-01:35872])
> 2012-12-19 11:12:12,162 WARN com.cloudera.flume.agent.MultiMasterRPC: Could
> not connect to any master nodes (tried 1: [flume-master-01:35872])
>
> from "top", I've seen a lot of java processes owned by flume. followed my
> instinct, I stopped flume agent, after a while, I run "top" on it, I still
> have a lot of processes owned by flume user.
>
> 12735 flume 20 0 1487m 26m 128 S 0.3 1.3 0:05.41 java
>
>
>
>
> 13828 flume 20 0 1487m 24 24 S 0.3 0.0 8:11.92 java
>
>
>
>
> 14442 flume 20 0 1487m 0 0 S 0.3 0.0 8:06.81 java
>
>
>
>
> 15033 flume 20 0 1487m 26m 164 S 0.3 1.3 0:04.29 java
>
>
>
>
> 15453 flume 20 0 1487m 4 4 S 0.3 0.0 8:07.55 java
>
>
>
>
> 15964 flume 20 0 1487m 27m 164 S 0.3 1.4 0:04.09 java
>
>
>
>
> 16098 flume 20 0 1487m 0 0 S 0.3 0.0 8:12.67 java
> ......................
>
> Why so many processes are generated here and they don't go away even when
> flume is stopped.
>
> --
> All the best,
> Shengjie Min
Re: flume master down, flume node still has multiple orphan processes
Posted by Shengjie Min <ke...@gmail.com>.
I'd love to move to 1.3.0, but unfortunately, this is the case on our
stage, we will need to live with it for a while.
Shengjie
On 19 December 2012 11:54, Nitin Pawar <ni...@gmail.com> wrote:
> first request : Please move to flume-1.3.0
>
>
>
>
>
> On Wed, Dec 19, 2012 at 5:04 PM, Shengjie Min <ke...@gmail.com>wrote:
>
>> Hi,
>>
>> I noticed that when my flume master is down for a quite while. My flume
>> agent is retrying to connect to the master, i can tell that from the flume
>> logs as below;
>>
>> 2012-12-19 11:11:42,126 WARN com.cloudera.flume.agent.MultiMasterRPC:
>> Could not connect to any master nodes (tried 1: [flume-master-01:35872])
>> 2012-12-19 11:11:52,140 WARN com.cloudera.flume.agent.MultiMasterRPC:
>> Could not connect to any master nodes (tried 1: [flume-master-01:35872])
>> 2012-12-19 11:12:02,152 WARN com.cloudera.flume.agent.MultiMasterRPC:
>> Could not connect to any master nodes (tried 1: [flume-master-01:35872])
>> 2012-12-19 11:12:12,162 WARN com.cloudera.flume.agent.MultiMasterRPC:
>> Could not connect to any master nodes (tried 1: [flume-master-01:35872])
>>
>> from "top", I've seen a lot of java processes owned by flume. followed my
>> instinct, I stopped flume agent, after a while, I run "top" on it, I still
>> have a lot of processes owned by flume user.
>>
>> 12735 flume 20 0 1487m 26m 128 S 0.3 1.3 0:05.41 java
>>
>>
>>
>>
>> 13828 flume 20 0 1487m 24 24 S 0.3 0.0 8:11.92 java
>>
>>
>>
>>
>> 14442 flume 20 0 1487m 0 0 S 0.3 0.0 8:06.81 java
>>
>>
>>
>>
>> 15033 flume 20 0 1487m 26m 164 S 0.3 1.3 0:04.29 java
>>
>>
>>
>>
>> 15453 flume 20 0 1487m 4 4 S 0.3 0.0 8:07.55 java
>>
>>
>>
>>
>> 15964 flume 20 0 1487m 27m 164 S 0.3 1.4 0:04.09 java
>>
>>
>>
>>
>> 16098 flume 20 0 1487m 0 0 S 0.3 0.0 8:12.67 java
>> ......................
>>
>> Why so many processes are generated here and they don't go away even when
>> flume is stopped.
>>
>> --
>> All the best,
>> Shengjie Min
>>
>>
>
>
> --
> Nitin Pawar
>
>
--
All the best,
Shengjie Min
Re: flume master down, flume node still has multiple orphan processes
Posted by Nitin Pawar <ni...@gmail.com>.
first request : Please move to flume-1.3.0
On Wed, Dec 19, 2012 at 5:04 PM, Shengjie Min <ke...@gmail.com> wrote:
> Hi,
>
> I noticed that when my flume master is down for a quite while. My flume
> agent is retrying to connect to the master, i can tell that from the flume
> logs as below;
>
> 2012-12-19 11:11:42,126 WARN com.cloudera.flume.agent.MultiMasterRPC:
> Could not connect to any master nodes (tried 1: [flume-master-01:35872])
> 2012-12-19 11:11:52,140 WARN com.cloudera.flume.agent.MultiMasterRPC:
> Could not connect to any master nodes (tried 1: [flume-master-01:35872])
> 2012-12-19 11:12:02,152 WARN com.cloudera.flume.agent.MultiMasterRPC:
> Could not connect to any master nodes (tried 1: [flume-master-01:35872])
> 2012-12-19 11:12:12,162 WARN com.cloudera.flume.agent.MultiMasterRPC:
> Could not connect to any master nodes (tried 1: [flume-master-01:35872])
>
> from "top", I've seen a lot of java processes owned by flume. followed my
> instinct, I stopped flume agent, after a while, I run "top" on it, I still
> have a lot of processes owned by flume user.
>
> 12735 flume 20 0 1487m 26m 128 S 0.3 1.3 0:05.41 java
>
>
>
>
> 13828 flume 20 0 1487m 24 24 S 0.3 0.0 8:11.92 java
>
>
>
>
> 14442 flume 20 0 1487m 0 0 S 0.3 0.0 8:06.81 java
>
>
>
>
> 15033 flume 20 0 1487m 26m 164 S 0.3 1.3 0:04.29 java
>
>
>
>
> 15453 flume 20 0 1487m 4 4 S 0.3 0.0 8:07.55 java
>
>
>
>
> 15964 flume 20 0 1487m 27m 164 S 0.3 1.4 0:04.09 java
>
>
>
>
> 16098 flume 20 0 1487m 0 0 S 0.3 0.0 8:12.67 java
> ......................
>
> Why so many processes are generated here and they don't go away even when
> flume is stopped.
>
> --
> All the best,
> Shengjie Min
>
>
--
Nitin Pawar