You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by "Rakesh Gande -X (rgande - CAPGEMINI INDIA PRIVATE LIMITED at Cisco)" <rg...@cisco.com> on 2015/04/06 15:01:26 UTC
Supervisor shutting down frequently
Hi All,
Supervisor in our clustered environment is dying frequently because of the following error. Any idea what can be causing this issue.
2015-04-06 18:13:51 o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-04-06 18:13:51 o.a.c.f.s.ConnectionStateManager [WARN] There are no ConnectionStateListeners registered.
2015-04-06 18:13:51 b.s.d.supervisor [INFO] Starting supervisor with id cb16a155-09a7-433f-b417-edb9b0bb51c4 at host <hostname>
2015-04-06 18:13:52 b.s.d.supervisor [INFO] Shutting down and clearing state for id 60fadbdc-2cfe-451a-bb81-e52e20955197. Current supervisor time: 1428324232. State: :disallowed, Heartbeat: #backtype.storm.daemon.common.WorkerHeartbeat{:time-secs 1427453846, :storm-id "CommonStreamTopology-19-1427356292", :executors #{[3 3] [5 5] [-1 -1] [1 1]}, :port 6703}
2015-04-06 18:13:52 b.s.d.supervisor [INFO] Shutting down cb16a155-09a7-433f-b417-edb9b0bb51c4:60fadbdc-2cfe-451a-bb81-e52e20955197
2015-04-06 18:13:52 b.s.event [ERROR] Error when processing event
java.io.IOException: . doesn't exist.
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:157) ~[commons-exec-1.1.jar:1.1]
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:147) ~[commons-exec-1.1.jar:1.1]
at backtype.storm.util$exec_command_BANG_.invoke(util.clj:378) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.util$ensure_process_killed_BANG_.invoke(util.clj:394) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.supervisor$shutdown_worker.invoke(supervisor.clj:175) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:240) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.applyToHelper(AFn.java:161) [clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na]
at clojure.core$apply.invoke(core.clj:619) ~[clojure-1.5.1.jar:na]
at clojure.core$partial$fn__4190.doInvoke(core.clj:2396) ~[clojure-1.5.1.jar:na]
at clojure.lang.RestFn.invoke(RestFn.java:397) ~[clojure-1.5.1.jar:na]
at backtype.storm.event$event_manager$fn__2378.invoke(event.clj:39) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
2015-04-06 18:13:52 b.s.util [INFO] Halting process: ("Error when processing an event")
2015-04-06 18:26:05 o.a.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
Thanks,
Rakesh
Re: Supervisor shutting down frequently
Posted by Richards Peter <hb...@gmail.com>.
Hi,
I was able to fix this issue on my setup. As Shivendra suggested in his
mail, it is the current working directory of supervisor process that causes
the issue.
The script that I use to start supervisor(wrapper script on
$STORM_HOME/bin/storm) switches from root user to storm user before the
supervisor is started. During the switching the current working directory
of supervisor process got set to /root due to a mistake that I committed in
my wrapper script.
In the wrapper script I tried to preserve the environment of root user
which made the current working directory of supervisor process to point to
/root. You can issue the following command after replacing <supervisor-pid>
with process id of supervisor to find the current working directory of
supervisor:
sudo ls -l /proc/<supervisor-pid>/cwd
After modifying the script to use the environment variables of storm user
during the switching, I found that the current work directory of supervisor
is set to /home/storm. This change in the script helped me to fix the issue.
Thanks,
Richards Peter.
Re: Supervisor shutting down frequently
Posted by Richards Peter <hb...@gmail.com>.
Hi,
I ran into this error today. I am using storm-0.9.5.
The issue was not caused due to directory permissions issue. I checked the
storm data directory and it had all the required permissions.
The issue occurred after running topologies for a while. I am not sure how
to debug the same.
Thanks,
Richards Peter.
Re: Supervisor shutting down frequently
Posted by Shivendra Singh <SS...@walmartlabs.com>.
Other thing I might check would be permission of storm data directory and current directory from where storm is being run.
However, if it is dying after running for a while, these would not be the reason.
Thanks,
Shivendra
From: Prajwal Tuladhar <pr...@infynyxx.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Monday, April 6, 2015 at 8:21 AM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Re: Supervisor shutting down frequently
Seems like due to a file or directory required in your topology code doesn't exist.
java.io.IOException: . doesn't exist.
On Mon, Apr 6, 2015 at 9:01 AM, Rakesh Gande -X (rgande - CAPGEMINI INDIA PRIVATE LIMITED at Cisco) <rg...@cisco.com>> wrote:
Hi All,
Supervisor in our clustered environment is dying frequently because of the following error. Any idea what can be causing this issue.
2015-04-06 18:13:51 o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-04-06 18:13:51 o.a.c.f.s.ConnectionStateManager [WARN] There are no ConnectionStateListeners registered.
2015-04-06 18:13:51 b.s.d.supervisor [INFO] Starting supervisor with id cb16a155-09a7-433f-b417-edb9b0bb51c4 at host <hostname>
2015-04-06 18:13:52 b.s.d.supervisor [INFO] Shutting down and clearing state for id 60fadbdc-2cfe-451a-bb81-e52e20955197. Current supervisor time: 1428324232. State: :disallowed, Heartbeat: #backtype.storm.daemon.common.WorkerHeartbeat{:time-secs 1427453846, :storm-id "CommonStreamTopology-19-1427356292", :executors #{[3 3] [5 5] [-1 -1] [1 1]}, :port 6703}
2015-04-06 18:13:52 b.s.d.supervisor [INFO] Shutting down cb16a155-09a7-433f-b417-edb9b0bb51c4:60fadbdc-2cfe-451a-bb81-e52e20955197
2015-04-06 18:13:52 b.s.event [ERROR] Error when processing event
java.io.IOException: . doesn't exist.
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:157) ~[commons-exec-1.1.jar:1.1]
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:147) ~[commons-exec-1.1.jar:1.1]
at backtype.storm.util$exec_command_BANG_.invoke(util.clj:378) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.util$ensure_process_killed_BANG_.invoke(util.clj:394) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.supervisor$shutdown_worker.invoke(supervisor.clj:175) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:240) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.applyToHelper(AFn.java:161) [clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na]
at clojure.core$apply.invoke(core.clj:619) ~[clojure-1.5.1.jar:na]
at clojure.core$partial$fn__4190.doInvoke(core.clj:2396) ~[clojure-1.5.1.jar:na]
at clojure.lang.RestFn.invoke(RestFn.java:397) ~[clojure-1.5.1.jar:na]
at backtype.storm.event$event_manager$fn__2378.invoke(event.clj:39) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
2015-04-06 18:13:52 b.s.util [INFO] Halting process: ("Error when processing an event")
2015-04-06 18:26:05 o.a.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
Thanks,
Rakesh
--
--
Cheers,
Praj
Re: Supervisor shutting down frequently
Posted by Prajwal Tuladhar <pr...@infynyxx.com>.
Seems like due to a file or directory required in your topology code
doesn't exist.
*java.io.IOException: . doesn't exist.*
On Mon, Apr 6, 2015 at 9:01 AM, Rakesh Gande -X (rgande - CAPGEMINI INDIA
PRIVATE LIMITED at Cisco) <rg...@cisco.com> wrote:
>
>
> Hi All,
>
> Supervisor in our clustered environment is dying frequently because of the
> following error. Any idea what can be causing this issue.
>
>
>
> 2015-04-06 18:13:51 o.a.c.f.s.ConnectionStateManager [INFO] State change:
> CONNECTED
>
> 2015-04-06 18:13:51 o.a.c.f.s.ConnectionStateManager [WARN] There are no
> ConnectionStateListeners registered.
>
> 2015-04-06 18:13:51 b.s.d.supervisor [INFO] Starting supervisor with id
> cb16a155-09a7-433f-b417-edb9b0bb51c4 at host <hostname>
>
> 2015-04-06 18:13:52 b.s.d.supervisor [INFO] Shutting down and clearing
> state for id 60fadbdc-2cfe-451a-bb81-e52e20955197. Current supervisor time:
> 1428324232. State: :disallowed, Heartbeat:
> #backtype.storm.daemon.common.WorkerHeartbeat{:time-secs 1427453846,
> :storm-id "CommonStreamTopology-19-1427356292", :executors #{[3 3] [5 5]
> [-1 -1] [1 1]}, :port 6703}
>
> 2015-04-06 18:13:52 b.s.d.supervisor [INFO] Shutting down
> cb16a155-09a7-433f-b417-edb9b0bb51c4:60fadbdc-2cfe-451a-bb81-e52e20955197
>
> 2015-04-06 18:13:52 b.s.event [ERROR] Error when processing event
>
> java.io.IOException: . doesn't exist.
>
> at
> org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:157)
> ~[commons-exec-1.1.jar:1.1]
>
> at
> org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:147)
> ~[commons-exec-1.1.jar:1.1]
>
> at backtype.storm.util$exec_command_BANG_.invoke(util.clj:378)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>
> at
> backtype.storm.util$ensure_process_killed_BANG_.invoke(util.clj:394)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>
> at
> backtype.storm.daemon.supervisor$shutdown_worker.invoke(supervisor.clj:175)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>
> at
> backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:240)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>
> at clojure.lang.AFn.applyToHelper(AFn.java:161)
> [clojure-1.5.1.jar:na]
>
> at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na]
>
> at clojure.core$apply.invoke(core.clj:619) ~[clojure-1.5.1.jar:na]
>
> at clojure.core$partial$fn__4190.doInvoke(core.clj:2396)
> ~[clojure-1.5.1.jar:na]
>
> at clojure.lang.RestFn.invoke(RestFn.java:397)
> ~[clojure-1.5.1.jar:na]
>
> at
> backtype.storm.event$event_manager$fn__2378.invoke(event.clj:39)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
>
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
>
> 2015-04-06 18:13:52 b.s.util [INFO] Halting process: ("Error when
> processing an event")
>
> 2015-04-06 18:26:05 o.a.z.ZooKeeper [INFO] Client
> environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
>
>
>
> Thanks,
>
> Rakesh
>
--
--
Cheers,
Praj